{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 线性回归到神经网络\n",
    "\n",
    "* 线性回归是最简单的神经网络形式。可以把神经网络看成是线性回归的一种扩展。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"14.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import Image\n",
    "Image(url= \"14.jpg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 线性回归的三种表现形式:方程，矩阵，神经网络图\n",
    "\n",
    "$$z = b + \\omega_1 x_1 + \\omega_2 x_2 + ... + \\omega_p x_p + \\epsilon$$\n",
    "\n",
    "其中:\n",
    "\n",
    "$z$ 是因变量,需要预测的目标\n",
    "$x_1, x_2, ..., x_p$ 是自变量,输入特征\n",
    "$b$ 是截距项\n",
    "$\\omega_1, \\omega_2, ..., \\omega_p$ 是回归系数\n",
    "$\\epsilon$ 是误差项\n",
    "\n",
    "模型的目标是找到一组参数,使得预测$\\hat{z}$尽可能接近真实观测值z。\n",
    "常见的训练方法是最小二乘法,通过最小化所有样本的平方误差来求解最优参数。\n",
    "\n",
    "多元线性回归广泛用于数值预测任务,可以看作是最简单的机器学习模型之一。但表达能力有限,只能表示线性关系。\n",
    "\n",
    "\n",
    "* 图解表达的好处\n",
    "\n",
    "* 线性回归本质上就是一个单层神经网络。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"output1.png\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "Image(url= \"output1.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"191.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url= \"191.jpg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 单层神经网络前向传播的代码实现\n",
    "\n",
    "* 简单代码实现\n",
    "\n",
    "torch.tensor\n",
    "torch.mv\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "x=torch.tensor([[1.0,2.0],[3.0,4.0],[5.0,6.0]])\n",
    "z=torch.tensor([2.0,3.0,5.0])\n",
    "w=torch.tensor([-0.6270,  0.6553])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def line0(x,w):\n",
    "    return torch.mv(x,w)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.6836, 0.7402, 0.7968])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line0(x,w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 通过torch.nn.Linear()函数构建隐藏层\n",
    "\n",
    "torch.nn.Linear\n",
    "output.weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.3280],\n",
       "        [0.3847],\n",
       "        [0.4414]], grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x=torch.tensor([[1.0,2.0],[3.0,4.0],[5.0,6.0]],dtype = torch.float32)\n",
    "output=torch.nn.Linear(2,1)\n",
    "output(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[-0.6270,  0.6553]], requires_grad=True)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output.weight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 激活函数（从线性回归到神经网络的第一个扩展）\n",
    "\n",
    "线性回归:\n",
    "\n",
    "$$ y = \\beta X + b $$\n",
    "\n",
    "其中,$X$是输入特征,$y$是输出,$b$是偏置项,$\\beta$是线性回归的参数。线性回归假设输入和输出之间是线性关系。\n",
    "\n",
    "单层神经网络:\n",
    "\n",
    "$$ y = f(X\\beta + b) $$\n",
    "\n",
    "其中$f$是一个非线性激活函数,如sigmoid。单层网络在线性回归基础上加了非线性变换。\n",
    "\n",
    "多层神经网络:\n",
    "\n",
    "$$ y = f_n(W_n...f_2(W_2f_1(W_1X+b_1)...)+b_n) $$\n",
    "\n",
    "其中$W_i, b_i$表示每一层的权重和偏置,$f_i$表示激活函数。多层网络堆叠线性变换和非线性变换,能够拟合更复杂的函数。\n",
    "\n",
    "从上式可以看出,线性回归是最简单的单层神经网络。随着层数的增加和非线性的引入,神经网络可以表示更加复杂的函数,拟合能力也越强。两者之间是渐进的关系。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"17.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url= \"17.jpg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 输出层的激活函数sigmoid——从回归到二分类\n",
    "\n",
    "\n",
    "\n",
    "Sigmoid函数将实数映射到(0,1)区间,是一个S型的曲线。它的定义形式为:\n",
    "\n",
    "$\\sigma(x) = \\frac{1}{1+e^{-x}}=\\frac{e^x}{e^x+1}$\n",
    "\n",
    "x为输入,$\\sigma(x)$为Sigmoid激活函数的值。\n",
    "\n",
    "Sigmoid函数的主要特点是:\n",
    "\n",
    "取值范围在(0,1)之间\n",
    "单调递增\n",
    "曲线S形,与x轴相交于(0,0.5)\n",
    "导数求解简单\n",
    "存在梯度消失问题\n",
    "Sigmoid函数广泛用于二分类神经网络中,将任意实数映射为0-1之间的概率。但目前由于梯度消失问题,在深层网络中已较少使用。\n",
    "\n",
    "\n",
    "* 在神经网络的输出层加入Sigmoid激活函数,主要有以下目的:\n",
    "\n",
    "1.将网络输出转换为概率\n",
    "Sigmoid函数将实数映射到(0,1)区间,可将网络输出转换为概率值,表示预测的置信度。这对于进行预测很有用。\n",
    "\n",
    "2.实现二分类输出\n",
    "Sigmoid的输出区间为(0,1),可以直接用来表示二分类任务中的概率P(y=1|x)。值接近0表示预测第一类,接近1表示预测第二类。\n",
    "\n",
    "3.定义binary cross entropy loss\n",
    "Sigmoid可与binary cross entropy loss配合使用,构建二分类任务的loss函数,用于训练网络模型。\n",
    "\n",
    "4.提供可微分性\n",
    "Sigmoid函数可导,使得输出层也可导,便于使用梯度下降进行网络参数优化。\n",
    "\n",
    "5.减少梯度消失\n",
    "相比tanh等激活函数,Sigmoid在正负饱和区的梯度更大,一定程度上减轻了梯度消失问题。\n",
    "\n",
    "6.提升收敛性\n",
    "Sigmoid可强制网络输出在一个数值范围内,增强网络的收敛性。\n",
    "\n",
    "总之,Sigmoid适合作为输出层激活函数,将网络输出转换为概率,定义loss函数,实现二分类,并一定程度上提升优化性能。但其梯度消失问题仍限制了网络层数的增长。\n",
    "\n",
    "torch.sigmoid\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "x=torch.tensor(range(-10,10),dtype = torch.float32)\n",
    "y=torch.sigmoid(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x17b48a76760>]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAApUAAAEYCAYAAAANlw08AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAAxOAAAMTgF/d4wjAAAyb0lEQVR4nO3deXxU9b3/8feZmSwkYcnCEkkmAWJkXwJBRSyKW6WIvaVW21KMgEur3lraq962dtFbirWNtfWnoAWUcvVaxVpbaVXUWqlVEgGh7AGSSTAQskBIQpaZ+f7+SIjETGAgy0kyr+fDPDJzznfm+5nDYXh7vud8j2WMMQIAAADawWF3AQAAAOj5CJUAAABoN0IlAAAA2o1QCQAAgHYjVAIAAKDdCJUAAABoN0IlAAAA2s1ldwEnRUREaODAgXaXAQAAgDYcOXJEdXV1Add1m1A5cOBAFRUV2V0GAAAA2pCUlNTmOoa/AQAA0G6ESgAAALQboRIAAADtRqgEAABAuxEqAQAA0G5Bhcr//M//VGpqqizL0pYtW9pst2LFCp1//vkaMWKEbr31VjU0NHRUnQAAAG0yxignv1wv5hYqJ79cxphe269dn/VMgppS6Mtf/rLuvfdeTZ8+vc02Bw4c0AMPPKBNmzZp8ODBuv766/XUU0/pzjvv7LBiAQBA8Iwxyi2oUH5ptVITojUlJVaWZfW6PosqajR/5UYVltcozOlQg8+v5LgorV4wVUmxUb2qX7s+azAscxbxNjU1Va+88oomTpzYat0jjzyiffv2admyZZKkdevWacmSJdqwYUNQ752UlMQ8lQAAdJBQCTzGGF2R/a4Kymrk838aaZwOS6nxUVq/eEanhFo7+rXrs57qdHmtwyY/93g8SklJaX6empoqj8fTUW8PAACCZIzR/JUbm8NHg88nSSooq9HNKze2O3z4/UZ1Xr/qvL7G3w1+1TZ4dcszuTp07IR8Rs195pdWa+6T7+uHXxglv5H8xsjnb3wPnzHy+U3TslMfK8CyxvZ+f8v1xZUnlF9aLf9nDpH5/Eb7S6v19d99qNjocMlIRkamqQZjJCOp8dDaqc+NjCS/UfOwsjnltScfV55o0IEj1frskTmf32j/kWpd8+g/FBPparH+5GE889kFpyz7tM0p65oeVtd5VVBWE7BPT3mNcgsqlJkaF8SfYOew7Y462dnZys7Obn5eVVVlVykAAHSqrhwS9vuN/r67RIXlLY9mSY3hI7+0Rveu3aqEmAjVNnwaCpsDotevugafapt+159c5vU1tfOr3ucPvh4jHa6s093Pb+ngT3pmxkjv7ytrtdxhSZZlyZJkWZIlS03/ybIkR/O6xt+frrOa2kv1Xn+rcHeqQ5W1iqr9NGad/OM+9U890D5w6qJPX9P4oKrO22Z/YU6H8kure0eodLvd2rdvX/Pz/Px8ud3uNtsvXrxYixcvbn5+utv+AADQU7V3SNgYo+p6n8qq6lRWXa+yqvrmx6VVdY3Pq0/+rld5dX2rMHkqnzF6MTfw8KXTYSnS5VBEmFMRLkfTj1MxkS5FupyKCPt0WYTL0fS8cfm+kmq9u6dEDb7WfYe7HPrKlGRNT4uXw7LkdFhyOCw5Tz5u+u106NP1zctOeWxZcjh0ymNLWwuP6vY1HwXsN8xpac3CC5WZGtcYBjswyOfkl+trT38QsF+X09KKrMwOD3in67PB51dqQnSH9ne2OixUzp07V9OnT9dPfvITDR48WMuWLdNNN93UUW8PAECPc7ph6K//7kM9/tVJnwbFU4LhqQGytKpOdd7THxmMiXApPiZcybF9NDF5gIwxentXSashYUlyOSwtnTtOU1LiPg2FTQHS5Tz3mQZz8sv17p6SNrfD9RPP65SjaJePHKTkuKiA5xm646I0dVhcpxwVnpISe9p+p6TE9oo+z0ZQF+rcfvvteu2113To0CHFx8erb9++ysvL06JFizRnzhzNmTNHkvT0009r6dKlkqTLLrtMy5YtU1hYWFCFcKEOAKC3+fvuEi16Nlfe0xw5DCTc5VBCdLjiYyIUHxOu+OgIJcSEKz4mXHHRjcsSmn7HRYcrMszZ4vWhdhFJoKPB7rgorV54oYYO6NMpfdrVr12f9aTT5bWzuvq7MxEqAQA9md9vtL+0SpsKjmqTp0KbPBXac7jt6wVcDktXjhqkGRcMUnxTgGwMjhGKDne2O4CFWuCxYyoju/q167NKhEoAADpcZW2DtnhOBsij2uKpUGXtpxdSnNc/UikJ0dq4v1y+AP/UhjktPXfrRZ16YUWoBR50vi6ZUggAgN4q0FHIvSVVzVO9hDsdGju0nzLcscpIiVWGO1ZD+keedki4K86BsyxLmalxXXpFsB19onsgVAIAQkawR9FOHoXc3HQkcnOAo5CzxiUqwx2rSe4BGnNeP0W4nK3ex7IsrV4wtc0hYY7goTchVAIAQkJbU/s8mzVVdT7fWR+FDFZSbJTeWjyDIWH0epxTCQDo9doahpYaJ8I+ddF5/SM1qSk8nu4oJBCKOKcSABDScgsqVFR+IuCk4MZIs8clatb4xLM+CgngU4RKAECv9/bOw/L6A08g3ifcqRkXDNSscYldXBXQuxAqAQC91u5Dx/WLv+3SW7sC3+lF6h63twN6A0IlAKDXKT52Qtlv7NHaTUXyG+kL44Zo28FjOni0tlve3g7oDQiVAIBe49iJBj3x9zw988981Xn9mjYiXvdfO1Ljkwac9m4vXIkNtB+hEgDQ49U2+PT7fxXo8XfydOxEg0YO6av7rx2pGekDmwMjU/sAnYtQCQDosfx+o1e2HNSv3tijg0dPaOiAPvrxdaN1/cShcjpah0Xu9gJ0HkIlAKDHMcbo3T1H9PDfdmtncaX69wnTD2aN0jcuTlFkGHNKAnYgVAIAepRtRcf087/u1Pv7yhTucuj2GcP1rRlp6h8VZndpQEgjVAIAeoSCsmr98o09+vPHn8iypBsmJ+k7V6XrvAF97C4NgAiVAIBurqyqTr99O0//+2GBGnxGM0cO0n2fH6kLhvS1uzQApyBUAgC6pZp6r1a8d0DL/7FfVXVeTUgeoP++dqQuGh5vd2kAAiBUAgC6Fa/PrxdyC/Xr9Xt15HidUuOj9PDc8Zo1bgjT/wDdGKESAGALY0yLOSMnuwfojR0l+sXru7T/SLUSYsL10PVjdNNUt8KcDrvLBXAGhEoAQJf77N1t6r1+OR2W6rx+RYU7dc+V52vRpcMVE8E/U0BPwd9WAECXMsZo/sqNKiirkc9v1ODzSZK8fqN+kS6tXzxDg/pF2lwlgLPFeAIAoEvlFlSoqPyEfH7Tat2JBp8KymtsqApAexEqAQBdKr+0Wo42/vUJczqUX1rdtQUB6BCESgBAl4qKcKq2wR9wXYPPr9SE6C6uCEBHIFQCALrMsZoGZb+xR5Lk+MzsQE6HJXdclKakxNpQGYD2IlQCALpEbYNPt67O1b4j1brr8jSlJkQrzGkpKtypMKel1PgorV54IXNRAj0UV38DADqd32/03T98rI355VpwyTB975oL9N2r01vMUzklJZZACfRghEoAQKdbsm6nXttWrFnjhuiHXxglSbIsS5mpccpMjbO5OgAdgeFvAECnWrHhgH634YAyU2OV/ZWJcnz2ZEoAvQKhEgDQaV7bWqz/eW2HRgyM1tPzpygyzGl3SQA6CaESANApNh4o13f+sEUJMRF65papGhAVbndJADoRoRIA0OH2Hj6uRc/mKMxhaVVWppLjouwuCUAn40IdAECHOlxZq6xVOaqp92llVqbGDu1vd0kAugChEgDQYY7XNuiWVTk6ePSEfnnDBH0ufaDdJQHoIgx/AwA6RIPPr2/97ybtKK7Ud69K15cnJ9ldEoAuRKgEALSbMUb3r92m9/aW6qtTk3XXzDS7SwLQxYIOlXv37tW0adOUnp6uzMxMbd++vVUbv9+vxYsXa/To0Ro/frwuv/xy5eXldWjBAIDuJ/vNPVq7qUgzRw7SQ9eP5c44QAgKOlTefvvtuu2227Rnzx7dd999ysrKatXm1Vdf1T//+U99/PHH2rp1q6644gp9//vf78h6AQDdzHMfevTbt/M0Iam/Hv/aJLmcDIIBoSiov/klJSXKzc3VvHnzJElz585VYWFhq6OQlmWprq5OtbW1MsaosrJSSUmcUwMAvdVbOw/rh69skzsuSiuyMhUVzvWfQKgK6m9/YWGhEhMT5XI1NrcsS263Wx6PR2lpn543c9111+mdd97RkCFD1LdvXw0dOlTvvvtu51QOALDVlsKjuuu5zerfJ0zPLpiqhJgIu0sCYKMOHaPIzc3Vv//9bx08eFCffPKJrrjiCt1xxx0B22ZnZyspKan5p6qqqiNLAQB0ooKyai18JkdGRiuyMjUsIdrukgDYLKhQmZycrOLiYnm9XkmNV/l5PB653e4W7VavXq2ZM2dqwIABcjgcuvnmm/XOO+8EfM/FixerqKio+ScmJqadHwUA0BXKqup088qNqqip12+/mqEMd6zdJQHoBoIKlYMGDVJGRobWrFkjSVq7dq2SkpJaDH1L0vDhw/X222+rvr5ekvSXv/xFY8eO7eCSAQB2OVHv08Jnc5VfVqOfXj9WV40ebHdJALqJoM+oXr58ubKysrRkyRL169dPq1atkiQtWrRIc+bM0Zw5c3TnnXdq586dmjBhgsLCwjRkyBAtW7as04oHAHQdn9/o7uc3a0vhUX3rshH6xkUpdpcEoBuxjDHG7iIkKSkpSUVFRXaXAQAIwBijB/70b635wKP/mDRU2V+ZwFyUQAg6XV5jMjEAwBk9+e4+rfnAo0vS4vXw3PEESgCtECoBAKf18qYi/eJvuzVySF89OW+ywl380wGgNb4ZAABt2rC3VPe+tFWJ/SP1zC1T1S8yzO6SAHRThEoAQEA7PqnUHWs+Up9wp55dMFVD+kfaXRKAboxQCQBo5eDRE7rlmY2q9/r11DemKH1wX7tLAtDNcZNWAEALx2oalLVyow5X1uk3X52ki0fE210SgB6AUAkAIc4Yo9yCCuWXVuu8AX302Po92ltSpR/MGqU5E86zuzwAPQShEgBCWFFFjeav3KjC8hqFOR060eCTMdKXM5K06NJhdpcHoAchVAJAiDLGaP7KjSooq5HPb9Tg80mSLEmbCyvsLQ5Aj8OFOgAQonILKlRUfkI+f8sbqxlJnvIa5RYQLAEEj1AJACEqv7RaLmfgO+OEOR3KL63u4ooA9GSESgAIUakJ0Wrw+QOua/D5lZoQ3cUVAejJCJUAEKKmpMRqYN+IVsudDkvuuChNSYm1oSoAPRUX6gBAiDJG6hsRJqlWYU5LYU6HGnx+ueOitHrhhbKswEPjABAIoRIAQtRLm4q0+/BxLZqeqmvGJiq/tFqpCdGakhJLoARw1giVABCCquq8euT13YqLDtfdV6Srf58wZabG2V0WgB6McyoBIAQ98U6ejhyv03evbgyUANBehEoACDGF5TX63YYDGjmkr27KdNtdDoBeglAJACHm53/dqXqvXz+aPVpOB+dOAugYhEoACCEf7C/Tum2HdPXowZqWlmB3OQB6EUIlAIQIn9/owT/vULjToR98YZTd5QDoZQiVABAiXswt1I7iSt0yPVUp8dwtB0DHIlQCQAg4XtugX76xWwkx4brr8jS7ywHQCxEqASAEPP5Onkqr6vW9qy9Q30imEALQ8QiVANDLFZRVa9WGfI1O7KcbpiTbXQ6AXopQCQC93JJ1O1Xv8+tH1zGFEIDOQ6gEgF7s/X2len37YV07doguGh5vdzkAejFCJQD0UqdOIfT9WUwhBKBzESoBoJd6IadQuw4d18JLhyk5LsrucgD0coRKAOiFKmsb9Ks3dmtg3wjdyRRCALoAoRIAeqHfvrVXZdX1+q9rLlBMhMvucgCEAEIlAPQyB0qr9cz7+Ro7tJ++nJFkdzkAQgShEgB6mZ+9tlMNPqMfzR4jB1MIAegihEoA6EU27C3V+p2H9YXxiZo6LM7ucgCEEEIlAPQSXp9fD/5lu8JdDv33tSPtLgdAiCFUAkAv8XxOofYcrtJtlw5XUixTCAHoWkGHyr1792ratGlKT09XZmamtm/fHrDdtm3bdNlll2nUqFEaNWqUXn755Q4rFgAQ2LGaBmW/sVuD+kbom5eNsLscACEo6Hkmbr/9dt12223KysrSSy+9pKysLOXk5LRoU1NTo+uvv16rV6/W9OnT5fP5VF5e3uFFAwBaeuytvaqoadCvbpigaKYQAmCDoI5UlpSUKDc3V/PmzZMkzZ07V4WFhcrLy2vR7rnnntNFF12k6dOnS5KcTqcGDhzYwSUDAE6170iVVv8rXxOS+us/Jg21uxwAISqoUFlYWKjExES5XI3/92tZltxutzweT4t2O3bsUEREhGbPnq2JEydq/vz5OnLkSMdXDQBo9rPXdsrrN/rRdaOZQgiAbTr0Qh2v16v169dr+fLl2rx5s4YOHapvfvObAdtmZ2crKSmp+aeqqqojSwGAkPDuniN6e1eJ5kw4T5NTmEIIgH2CCpXJyckqLi6W1+uVJBlj5PF45Ha7W7Rzu926/PLLNXToUFmWpXnz5umDDz4I+J6LFy9WUVFR809MTEw7PwoAhJYGn18P/WWHIsMcup8phADYLKhQOWjQIGVkZGjNmjWSpLVr1yopKUlpaWkt2n3lK19RTk6OKisrJUnr1q3ThAkTOrhkAIAkPfehR3klVbrtcyN03oA+dpcDIMQFfYng8uXLlZWVpSVLlqhfv35atWqVJGnRokWaM2eO5syZI7fbre9///uaNm2aHA6Hhg4dqqeeeqrTigeAUHW0pl6Prt+jIf0idceM4XaXAwCyjDHG7iIkKSkpSUVFRXaXAQA9wk9e3a5n3s/Xr2+cqC9yxTeALnK6vMYddQCgh8krOa7ff1CgickDNGfCeXaXAwCSCJUA0OM89Jed8vmNfswUQgC6EUIlAPQg7+wq0bt7jug/Jg3VJHes3eUAQDNCJQD0EA0+vx56bYf6hDl17+cvsLscAGiBUAkAPcTv/1Wg/UeqdceMEUrszxRCALoXQiUA9ADl1fX69fo9Oq9/pG77HFMIAeh+CJUA0AM8+uYeVdZ6df+sUeoT7rS7HABohVAJAN3c7kPH9b8fFmhySqyuG59odzkAEBChEgC6MWOM/ue1HfIb6UezR8uymEIIQPdEqASAbuytnSV6b2+p5mYkaULyALvLAYA2ESoBoJuq9/r1s3U7FRXOFEIAuj9CJQB0U6v/la8DpdX61mUjNLhfpN3lAMBpESoBoBsqq6rTY2/t1dABfbToUqYQAtD9uewuAADQyBij3IIK5ZdW640dh3S81qulXxqvyDCmEALQ/REqAaAbKKqo0fyVG1VYXiOnw1Jtg1+RLofGJ/W3uzQACArD3wBgM2OM5q/cqIKyGjX4jGob/JIa7/WdtWqjjDE2VwgAZ0aoBACb5RZUqKj8hHz+luHRZyRPeY1yCypsqgwAgkeoBACb5ZdWy+UMPKl5mNOh/NLqLq4IAM4eoRIAbJaaEK0Gnz/gugafX6kJ0V1cEQCcPUIlANhsSkqszhvQp9Vyp8OSOy5KU1JibagKAM4OoRIAbGZZliY0XeXtdFiKCncqzGkpNT5KqxdeyP2+AfQITCkEADbb/skx/Xlrsaamxup711yggrIapSZEa0pKLIESQI9BqAQAGxlj9OCfd0iSfjxnjMac119Th8XbXBUAnD2GvwHARq9vP6QPD5TrpsxkjTmPic4B9FyESgCwSW2DTz9bt1MxES4tvuoCu8sBgHYhVAKATVb+84AKy0/o7plpGtg3wu5yAKBdCJUAYIOS47X6f2/nKSU+SlmXpNpdDgC0GxfqAIANfvn6blXX+5Q9a5QiXE67ywGAduNIJQB0sX8fPKYXPyrStBHxunr0YLvLAYAOQagEgC50cgohS9KPrhvNPJQAeg1CJQB0oXXbDmljfrm+OtWtkUP62V0OAHQYQiUAdJHaBp+WrNupvpEuLb4q3e5yAKBDESoBoIv87r39Onj0hL59xfmKj2EKIQC9C6ESALrA4cpaPfH3fRqWEK35F6faXQ4AdDhCJQB0gV/8bbdq6n36waxRCnfx1Qug9+GbDQA62ceFR7V2U5EuPT9BV4waZHc5ANApgg6Ve/fu1bRp05Senq7MzExt3769zbbGGM2cOVMDBgzoiBoBoMcyxujBv+yQw5IemM0UQgB6r6BD5e23367bbrtNe/bs0X333aesrKw22z766KMaMWJER9QHAD3an7cW66OCCn39whSlD+5rdzkA0GmCCpUlJSXKzc3VvHnzJElz585VYWGh8vLyWrXdvn27XnnlFd1///0dWykA9DAn6n1aum6n+kW69B2mEALQywUVKgsLC5WYmCiXq/FW4ZZlye12y+PxtGjX0NCgW2+9VcuXL5fTefp72WZnZyspKan5p6qq6hw/AgB0T0+/t1+fHKvVPVemKy463O5yAKBTdeiFOj/96U/1pS99SaNGjTpj28WLF6uoqKj5JyYmpiNLAQBbHTpWqyf/vk/DB0brGxen2F0OAHQ6VzCNkpOTVVxcLK/XK5fLJWOMPB6P3G53i3bvvvuuPB6PHn/8cXm9XlVWVio1NVU5OTkaOHBgp3wAAOiOHv7bLp1o8OmBL4xWmJOJNgD0fkF90w0aNEgZGRlas2aNJGnt2rVKSkpSWlpai3bvvfeeCgoKlJ+frw0bNqhfv37Kz88nUAIIKZs9Ffrj5oOakT5Ql49kCiEAoSHo/31evny5li9frvT0dC1dulSrVq2SJC1atEivvvpqpxUIAD3JySmEnA5LD8w+86lAANBbWMYYY3cRkpSUlKSioiK7ywCAdnll80Hd88IWZU1L1U/mjLG7HADoUKfLa5zoAwAdpKbeq6V/3aUBUWG658rz7S4HALoUoRIAOsjyd/frUGWtvnNlugZEMYUQgNBCqASADvDJ0RNa/o99ShsUo69d6D7zCwCglyFUAkAHePhvu1Tb4NcDs5lCCEBo4psPANrpo4IK/WnLJ5o5cpBmpDOFGoDQRKgEgHbw+40e/PN2uRyWfvAFphACELoIlQDQDn/cfFAfFx3T/ItTNWIgt5sFELoIlQBwjqrrvPrF67sUGxWmb1/BFEIAQhuhEgDO0bJ39+lwZZ0WX32B+keF2V0OANiKUAkA56CookZP/WO/LhjcV1/NTLa7HACwHaESAM7B0r/uUp23cQohF1MIAQChEgDOVk5+uf6ytVhXjhqs6ecn2F0OAHQLhEoAOAuNUwjtUJiTKYQA4FSESgA4C2s3FWnbwWPKmpaqYQnRdpcDAN0GoRIAglRV59UvXt+t+Ohw3c0UQgDQAqESAIL0xDt5OnK8Tt+9+gL1i2QKIQA4FaESAIJQWF6j3204oJFD+upGphACgFYIlQAQhJ//dafqvX796LrRcjosu8sBgG6HUAkAZ/DB/jKt23ZI14wZrGkjmEIIAAIhVALAafj8Rg/9ZYfCnQ59fxZTCAFAW1x2FwAA3ZExRrkFFfpDTqG2f1Kp2z43TCnxTCEEAG0hVALAZxRV1Gj+yo0qLK9Rg89Ikt7YfljzL05VUmyUzdUBQPfE8DcAnMIYo/krN6qg7NNAKUmFFSd088qNMsac5tUAELoIlQBwityCChWVn5DP3zI8+vxGnvIa5RZU2FQZAHRvhEoAOEV+abWcbXwzhjkdyi+t7tqCAKCHIFQCwClio8J1osEfcF2Dz69U7vcNAAERKgGgSU29V795e68k6bPzmzsdltxxUZqSEmtDZQDQ/REqAUCS1+fX3c9t1taiY/rGxW6lJkQrzGkpKtypMKel1PgorV54oSyLu+kAQCBMKQQg5Blj9KNXt+utXSX6UsZQPThnrKTGi3byS6uVmhCtKSmxBEoAOA1CJYCQ98Tf9+m5Dz2anpagpV8a3xweM1PjlJkaZ3N1ANAzMPwNIKSt/ahIj7y+W6MS++nJeRkKd/G1CADngm9PACHrvb1HdN/arTqvf6SeuSVTfSPD7C4JAHosQiWAkLT9k2P65ppNigp36tkFUzW4X6TdJQFAj8Y5lQBCTlFFjW5ZlaN6r1+/XzhV5w/ua3dJANDjESoBhJRjNQ3KWpWjkuN1evxrk3Th8Hi7SwKAXiHo4e+9e/dq2rRpSk9PV2ZmprZv396qzdtvv62pU6dq9OjRGjNmjO699175/YHvTAEAXa22wadbf5+rvJIq/fALozR7/Hl2lwQAvUbQofL222/Xbbfdpj179ui+++5TVlZWqzaxsbH6v//7P+3YsUMfffSR3n//fa1evboj6wWAc+L3G333xY+18UC5FlwyTIsuHW53SQDQqwQVKktKSpSbm6t58+ZJkubOnavCwkLl5eW1aDdp0iQNH974RR0ZGamJEycqPz+/YysGgHOwZN1Ovba1WLPGDdEPvzDK7nIAoNcJKlQWFhYqMTFRLlfjKZiWZcntdsvj8bT5mkOHDumll17S7NmzO6ZSADhHKzYc0O82HFBmaqyyvzJRjs/e2BsA0G6dMqVQZWWlrrvuOt17772aMmVKwDbZ2dlKSkpq/qmqquqMUgCEuHXbivU/r+3QiIHRenr+FEWGOe0uCQB6JcsYY87UqKSkRGlpaSovL5fL5ZIxRomJidqwYYPS0tJatD1+/LiuueYazZo1Sz/84Q+DLiQpKUlFRUVn/wkAoA0bD5Rr3ooP1b9PmF7+5jQlx0XZXRIA9Giny2tBHakcNGiQMjIytGbNGknS2rVrlZSU1CpQVlVV6fOf/7w+//nPn1WgBICOlldyXLeuzlWYw9KqrEwCJQB0sqCHv5cvX67ly5crPT1dS5cu1apVqyRJixYt0quvvipJeuyxx7Rx40a9/PLLmjhxoiZOnKif/exnnVM5ALThcGWtbl6Zo+o6r56cN1ljh/a3uyQA6PWCGv7uCgx/A+gIVXVefWXZv7SjuFKPfHm8bpiSbHdJANBrtHv4GwB6ggafX99c85F2FFfqu1elEygBoAsRKgH0CsYY3b92m97bW6qvTk3WXTPTzvwiAECHIVQC6BWy39yjtZuKNHPkID10/VhZFnNRAkBXIlQC6PGe+9Cj376dp/FJ/fX41ybJ5eSrDQC6Gt+8AHq0t3Ye1g9f2SZ3XJRWZmUqKtxld0kAEJIIlQB6rI8Lj+qu5zarf58wPbtgqhJiIuwuCQBCFqESQI9UUFatBc/kyG+MVmRlalhCtN0lAUBII1QC6HHKqup088qNqqip12+/OkkZ7li7SwKAkMfJRwB6lBP1Pi18Nlf5ZTV66ItjdfWYIXaXBAAQoRJAN2eMUW5BhfJLq+WOi9JT/9ivLYVH9c3LRugbF6XYXR4AoAmhEkC3VVRRo/krN6qwvEZhTodqG3zyG+nq0YP0X1dfYHd5AIBTECoBdEvGGM1fuVEFZTXy+Y0afL7mdXlHqsXc5gDQvXChDoBuKbegQkXlJ+Tzm1brCstrlFtQYUNVAIC2ECoBdEv5pW0fjQxzOpRfWt21BQEATotQCaDb2Vlcqf/90KM6rz/g+gafX6nMSwkA3QrnVALoNg4ePaFfvbFbf9x8UMZIMREunaj3yWc+HQJ3Oiy546I0JYW5KQGgOyFUArDd0Zp6PfH3fXrm/XzVe/2anpag+68dqQFRYS2u/m7w+eWOi9LqhRfK4kodAOhWCJUAbFPb4NMz7+friXfyVFnr1Zjz+un+a0fq0vMHNrd5a/GM5nkqUxOiNSUllkAJAN0QoRJAl/P5jV7eVKTsN/eo+FitkmL76MHrx2rOhPPkcLQMjJZlKTM1TpmpcTZVCwAIBqESQJcxxuid3SV6+K+7tfvwccVGhemB2aM17yK3IlxOu8sDALQDoRJAl9hSeFQ/X7dTHx4oV2SYQ9+6bITuuGyE+kWG2V0aAKADECoBdKoDpdV65PVdWrftkByWdOOUZH3nqnQN6R9pd2kAgA5EqATQKY4cr9Nv3tqr5zd65PUbXTlqsO77/AU6f3Bfu0sDAHQCQiWADlVV59Xv3tuvp/+xX9X1PmW4B+i/Z43iQhsA6OUIlQA6RIPPr//b6NFjb+1VaVW9hg+M1r3XjNQ1YwYzBRAAhABCJYCgGGMCzhdpjNFf/31Ij7y+WwdKqzWwb4R+9h9jdeOUZLmc3AkWAEIFoRLAGRVV1LS6s01yXJS+c2W6frfhgD4uPKrocKcWX5WuRZcOU1Q4Xy0AEGr45gdwWsYYzV+5UQVlNfL5jRp8PknSgSPVuvv5zXI5pKxpqbprZpoSYiJsrhYAYBdCJYDTyi2oUFH5Cfn8psVyI8mypEdvnKjrJgy1pzgAQLdBqATQijFGB0qrtclzVC99VKgGvz9guz5hTtU2BF4HAAgthEoAqq7z6uPCo9rkqdAmz1Ft9lSooqbhjK9r8PmVmhDdBRUCALo7QiUQYk49CrnJU6HNnqPafahSJ0e3w5yWRp/XX1+cNEAZ7lhNcg/Q/BUbVVBe02II3Omw5I6L0pSUWJs+CQCgOyFUAj1QW9P7BHKmo5CD+kbo6tFDlJHSGCLHDu2vyDBni/dYvXBqq6u/3XFRWr3wQuagBABIkixjjDlzs86XlJSkoqIiu8sAur22pvdZvWCqhg7oo/yyGm0qqGgOkYGOQk5KHqCMlFhluAdo6IA+QQXDswmyAIDe6XR5jVAJ9CDGGF2R/W7z9D4nWVbjRTPhTktHT3iblw/qG6EMd+xpj0ICABCs0+U1hr/RK9h1FK0z+jXGqKrOq/LqepVW1ausqk5l1Y2/dxRXKr+0Wp+Z3UfGSDX1PiUOjNYXJyWd9VFIAADaK+hQuXfvXt18880qLS1V//799cwzz2jMmDGt2q1YsUJLly6V3+/XzJkz9cQTTygsLKxDi24PO8JHbwo83bHf0w0HJ8VGdYt+axt8Kq+uV1lVvUqr61TWIizWq+yUZaXV9ar3nv00PVHhTt0xY4RumJLcUR8RAICgBT38PXPmTM2fP19ZWVl66aWX9PDDDysnJ6dFmwMHDuiSSy7Rpk2bNHjwYF1//fW65pprdOedd57x/bti+NuO8NETAk9P7ret4WCnw1JqfJTWL57RrkBrjFGd19/40+BreuzTiXqfbl/zkQ4dq21x1NCypL4RLk0dFtcYIptCY1Wdt+1OJEW4HEqIiVB8TLjio8MV3/Q4Ibrxd1x0uBJiIlRUUaO7n9+sBl/rv7ZhTkvP3XqRMlPjzvnzAgBwOu0+p7KkpERpaWkqLy+Xy+WSMUaJiYnasGGD0tLSmts98sgj2rdvn5YtWyZJWrdunZYsWaINGza0q8iO0Nnho7v02V37dcdF6Y/fnCa/JJ/fyG+MfH7T4nHj78+sN0Z+/6mP1WLZ7sPH9eibe+T97HiwJKdl6cuTkxQfE94cBusa/M2Paxualnn9TcsbH5+6/FyOGEqSw1JTSIxoConhim8KiAkx4YqLbhkao8KdQV8sY8efLQAAUgecU1lYWKjExES5XI3NLcuS2+2Wx+NpESo9Ho9SUlKan6empsrj8bSn9g7T1q3mfH6j/UeqNeORdxQVfubNcboIbtRy5Yl6n4oqTuizL/H5jfYdqdb0h99RZJij6bUt3qjVspPZv+Wy1v0aI9V5fTpyvL5VfSf7nfTgmwpzOZpeb2RM4/v6TdNj0/SOTctPPjemqU3zuk9fa4xpdZ7fqf0eKK3WxIfeDNygk/iM0Qu5hQHXWZYU6XIqMsyhCJdTEWEORbgcio0KV4TLocgwpyJcjqblzpbLXA7tLanS+p2HAx4x7BPm1E/njNFXMjt+GNqyLK1ewPQ+AIDux7YLdbKzs5Wdnd38vKqqqlP7yy+tlstpqd4XeH1NvU9OR2PAO+M/y6dpcOqqWm8bnakx1Hh9fjkjPr0S12p6daBccDIsWM3P1artyddX1NTLYSlgyHNYUv+oMA2MiZBlNb3GalxuyWpcdsrjk31bTa89+bhxXVP7pucHK05o56HjrYK71HgkbXJKrNIGxchpWXI6LDksS06H5HBYn1n2mfWtljW2dzgsFZRV6Ym/7w/Yp8th6edfGqfJKbGKCHMq0uVQRFMwdDmsdgWwnPxyrd95OOA6r9+vYQM77y4zSbFRemvxDKb3AQB0K0GFyuTkZBUXF8vr9TYPf3s8Hrnd7hbt3G639u3b1/w8Pz+/VZuTFi9erMWLFzc/T0pKOpf6g5aaEK0GX+ChTJfT0pPzJnf4uWg5+eX62tMfBDya5XJYevzrGZ1y/tvJfv0B+nU6LP3yhgmd2m+gKO2wpP+65oIO79cYo9e2HQo4HJwSH6UvT07qlLA1JSVWyXFRbQ71d/ZdZizLUmZqHOdPAgC6DUcwjQYNGqSMjAytWbNGkrR27VolJSW1GPqWpLlz5+rVV1/VoUOHZIzRsmXLdNNNN3V81efgZAhwOloGjM4MAXb0GWr9nhwOTomPUpjTUlS4U2HOxvMLO3M42K5+AQDoroK++nv37t3KyspSWVmZ+vXrp1WrVmncuHFatGiR5syZozlz5kiSnn76aS1dulSSdNlll2nZsmVBTSlk19XfJ89FGzqgT6/pMxT7DZXpkwAAsBN31DkF81T23n4BAEDnIlQCAACg3U6X14I6pxIAAAA4HUIlAAAA2o1QCQAAgHYjVAIAAKDdCJUAAABot25z9XdERIQGDhzYZf1VVVUpJiamy/rrqdhOwWE7BYftFBy2U/DYVsFhOwWH7XRmR44cUV1dXcB13SZUdjWmMAoO2yk4bKfgsJ2Cw3YKHtsqOGyn4LCd2ofhbwAAALQboRIAAADtFrKhcvHixXaX0COwnYLDdgoO2yk4bKfgsa2Cw3YKDtupfUL2nEoAAAB0nJA9UgkAAICOQ6gEAABAu/XaUPnaa69p8uTJioiI0D333NNind/v1913360RI0YoLS1Njz/+eJvvs3fvXk2bNk3p6enKzMzU9u3bO7ly+9x5552aOHFi809kZKR+85vfBGx72WWXadiwYc1tH3300S6u1l4/+clPNHDgwObP//Wvf73NtqG0D33Wb37zG40dO1bjxo3T+PHjtWbNmjbbhto+Fex+sWLFCp1//vkaMWKEbr31VjU0NHRxpfaqra3VF7/4RaWnp2vChAm66qqrlJeX16pdfn6+nE5ni++wffv22VCxfVJTU3XBBRc0f/4XXnghYLtQ3qfKyspa7CPp6elyuVwqLy9v0Y796RyZXmr37t1my5Yt5gc/+IH59re/3WLds88+a2bOnGm8Xq8pKyszbrfb/Pvf/w74PpdffrlZtWqVMcaYF1980UyZMqWTK+8eiouLTWRkpCkuLg64fsaMGeaPf/xj1xbVjfz4xz9utV+1JVT3IWOMWb9+vTl69KgxxhiPx2Pi4+NNXl5ewLahtk8Fs1/s37/fJCYmmuLiYuP3+811111nHn/88S6u1F4nTpwwr732mvH7/cYYY37729+aGTNmtGp34MAB079//64trptJSUkxmzdvPm0b9qmWHnnkETN79uxWy9mfzk2vPVJ58v9qXS5Xq3UvvPCCbr31VjmdTsXFxenGG2/U888/36pdSUmJcnNzNW/ePEnS3LlzVVhYGPD/knubZ599Vtdcc42GDBlidyk9WijvQ5J0xRVXqH///pKk5ORkDRkyRIWFhTZXZb9g94uXXnpJc+bM0ZAhQ2RZlu64446A31W9WWRkpGbNmiXLsiRJF110kfLz8+0tqgdjn2ppxYoVWrhwod1l9Bq9NlSejsfjUUpKSvPz1NRUeTyeVu0KCwuVmJjYHEwty5Lb7Q7YtrdZuXLlGf+i3X///Ro3bpxuvPFG7d+/v4sq6z5efPFFTZgwQTNnztQ777wTsE0o70OftX79elVUVCgzM7PNNqGyTwW7XwT7XRVKHnvsMV1//fUB11VXVyszM1MZGRl68MEH5fP5urg6+82fP1/jxo3TwoULdeTIkVbr2ac+9f7776uiokKzZ88OuJ796ez12FB58cUXKyEhIeAPR0ICC3abvffeezp+/LhmzZrV5nv9/ve/165du7R161Zdeumlbf6l7KnOtK3uuOMO5efn6+OPP9ZDDz2kG2+8UQUFBXaX3eWC3ae2bdumW265RS+88IKio6MDvldv36fQfkuWLFFeXp5+/vOft1qXmJiogwcPKicnR+vXr9d7772nX/3qVzZUaZ9//OMf2rp1qzZt2qSEhATdfPPNdpfUra1YsULz588POKLJ/nRuWm/JHuJf//rXOb/W7XaroKBAF198saTGE3LdbnerdsnJySouLpbX65XL5ZIxRh6PJ2DbniDYbbZixQrdfPPNcjqdbbZJTk6W1HiE5a677tL3vvc9lZWVKT4+vkNqtdvZ7F+XXHKJJk2apNzc3BZHAKTetw99VjDbaceOHZo9e7ZWrlyp6dOnt9mut+9Tpwp2v3C73S0uDmjruyoU/PKXv9TLL7+s9evXKyoqqtX6iIgIDRo0SJIUFxenBQsW6LnnntO9997b1aXa5uS+ERYWpnvuuUfp6ekB27BPSVVVVfrDH/6gnJycgOvZn85Njz1S2R433HCDnn76afl8PpWXl+uFF17QjTfe2KrdoEGDlJGR0XzF6tq1a5WUlKS0tLSuLrnLVFZW6qWXXtKCBQvabOP1enX48OHm52vXrtXgwYN75T/+bSkqKmp+vHfvXm3ZskXjxo1r1S4U96FT7dy5U7NmzdJTTz2lq666qs12obZPBbtfzJ07V6+++qoOHTokY4yWLVumm266yY6SbZWdna3nn39eb775pgYMGBCwTUlJSfNVzHV1dXr55Zc1adKkLqzSXtXV1Tp69Gjz8+effz7g52efavTCCy9owoQJGjlyZMD1ob4/nTNbLxPqROvXrzdDhw41ffv2NTExMWbo0KHmT3/6kzHGGK/Xa771rW+ZYcOGmeHDh5tf//rXza/705/+ZBYuXNj8fNeuXeaiiy4y559/vpk8ebLZunVrl3+WrrR8+XLzuc99rtXynJwcc+211xpjjKmqqjKTJ082Y8eONePHjzczZ840W7Zs6epSbTV//nwzZswYM2HCBJORkWFefPHF5nWhvg+d6sorrzQDBgwwEyZMaP7529/+Zoxhn2prv1i4cGHzd5Uxxjz11FNm+PDhZvjw4WbBggWmvr7erpJtUVhYaCSZ4cOHN+9DU6dONcYY88ADD5gnn3zSGGPM2rVrzZgxY8z48ePN6NGjzV133WVqa2vtLL1L7du3z0ycONGMGzfOjB071syZM8ccOHDAGMM+FcjFF19sVq5c2WIZ+1P7cZtGAAAAtFtIDn8DAACgYxEqAQAA0G6ESgAAALQboRIAAADtRqgEAABAuxEqAQAA0G6ESgAAALQboRIAAADtRqgEAABAu/1/UgzwUuKTOEIAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 800x320 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize = (10,4),dpi = 80)\n",
    "plt.plot(x,y,\"o-\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 单层二分类神经网络的前向过程的代码实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1., 1., 1.])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x=torch.tensor([[1.0,2.0],[3.0,4.0],[5.0,6.0]])\n",
    "#z=torch.tensor([0,0,1])\n",
    "w=torch.tensor([2.3,1.4])\n",
    "\n",
    "def line1(x,w):\n",
    "    h1=torch.mv(x,w)\n",
    "    h1=torch.sigmoid(h1)\n",
    "    output=torch.tensor([int(x) for x in h1 >=0.5],dtype=torch.float32)\n",
    "    return output\n",
    "\n",
    "line1(x,w)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0., 1., 1.])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x=torch.tensor([[1.0,2.0],[3.0,4.0],[5.0,6.0]])\n",
    "z=torch.tensor([0,0,1])\n",
    "\n",
    "h1=torch.nn.Linear(2,1)\n",
    "h1_out=h1(x)\n",
    "h1_s_out=torch.sigmoid(h1_out)\n",
    "h1_s_c_out=torch.tensor([int(x) for x in h1_s_out >=0.5],dtype=torch.float32)\n",
    "h1_s_c_out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"17.png\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url= \"17.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"18.png\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "Image(url= \"18.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 神经元与隐藏层的增减（从线性回归到神经网络的第二个扩展）\n",
    "\n",
    "* 神经元与隐藏层的增减的效果（chatgpt的回答）\n",
    "\n",
    "在构建深度学习模型时,增加单层神经元数量和增加隐藏层数量这两种方法都可以增强网络的表达能力,但带来的效果有些差异:\n",
    "\n",
    "1. 增加单层神经元数量,主要作用是提升该层对更多特征的学习表达能力。适合当前层特征不够情况。\n",
    "\n",
    "2. 增加隐藏层数量,主要作用是堆叠更多的非线性变换,提升全网络的层次化特征学习能力。适合功能表达不足情况。\n",
    "\n",
    "3. 增加单层神经元数量对网络宽度产生影响,而增加隐藏层对网络深度产生影响。\n",
    "\n",
    "4. 单层神经元增多容易导致过拟合;而层数增多可以通过有效的预训练及正则化方法减轻过拟合。\n",
    "\n",
    "5. 层数增多会使得训练时间加长,而单层神经元增加影响较小。\n",
    "\n",
    "综上,增多神经元数量的效果局限在单层,而增加层数影响全网;当特征表示力不足时,可以适当加大单层宽度;当层次表示不够时,可以适当加深网络。需要根据具体情况选择最优的网络结构。\n",
    "\n",
    "* 各层之间的衔接以及输出层\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"19.png\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url= \"19.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 输出层的softmax函数-从回归到多分类\n",
    "\n",
    "对于一个向量x = [x1, x2, ..., xn],其softmax函数定义为:\n",
    "\n",
    "$$ softmax(x_i) = \\frac{e^{x_i}}{\\sum_{j=n} e^{x_j}} $$\n",
    "\n",
    "即对每个元素xi:\n",
    "\n",
    "$$ softmax(x_i) = \\frac{e^{x_i}}{e^{x_1} + e^{x_2} + ... + e^{x_n}}$$\n",
    "\n",
    "softmax函数的主要性质是:\n",
    "\n",
    "将任意实数向量归一化为概率分布,范围在[0, 1]\n",
    "概率和为1,即$\\sum_{i=1}^n softmax(x_i) = 1$\n",
    "保留了输入元素相对大小的关系\n",
    "\n",
    "softmax函数通常被用在神经网络的输出层,具有以下目的和意义:\n",
    "\n",
    "    1.将网络输出转换为概率分布,表示对各类别的预测概率。\n",
    "\n",
    "    2.实现多分类任务,将类别预测问题建模为概率估计问题。\n",
    "\n",
    "    3.定义多分类交叉熵损失函数,用于模型训练。\n",
    "\n",
    "    4.提供预测结果,选择概率最大的类别作为预测输出。\n",
    "\n",
    "    5.增加模型确定性,生成值域范围固定的概率输出。\n",
    "\n",
    "    6.提供类别之间可比性,概率表示预测的置信度。\n",
    "\n",
    "综上,softmax可将网络输出转换为概率,实现多分类,定义损失函数,最终用于类别预测,这是它被广泛应用于神经网络输出层的主要原因。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([9.9995e-01, 4.5398e-05, 2.0611e-09])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z=torch.tensor([1010,1000,990],dtype=torch.float32)\n",
    "torch.softmax(z,dim=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.0900, 0.2447, 0.6652],\n",
       "        [0.9362, 0.0171, 0.0466]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.tensor([[1, 2, 3], [9, 5, 6]], dtype=torch.float32)\n",
    "torch.softmax(x, dim=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"16.png\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url= \"16.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 激活函数（隐藏层）\n",
    "\n",
    "* 隐藏层的激活函数主要有以下几个作用（chatgpt）:\n",
    "\n",
    "1. 引入非线性:\n",
    "激活函数的引入让网络可以拟合复杂的非线性函数,而不仅是简单的线性映射,这大大提高了模型的表达能力。\n",
    "\n",
    "2. 定义输出范围:\n",
    "不同的激活函数将输出值映射到不同的范围,如Sigmoid将实数映射到[0,1],tanh映射到[-1,1]。这为后续层提供合适数值范围的输入。\n",
    "\n",
    "3. 实现层间转换:\n",
    "激活函数实现了层与层之间的转换,一个层的输出通过激活函数成为下层的输入,完成信息的传递和转换。\n",
    "\n",
    "4. 提供可训练参数:\n",
    "激活函数的参数也可以加入训练,如批归一化层中的可学习参数,提高模型的表达能力。\n",
    "\n",
    "5. 提高泛化能力:\n",
    "激活函数增强了模型对未见样本的泛化能力,如ReLU具有稀疏化作用,减少了过拟合风险。\n",
    "\n",
    "6. 提供梯度:\n",
    "激活函数可导,为网络的反向传播提供了梯度,使网络可以通过梯度下降进行训练。\n",
    "\n",
    "7. 提供正则化:\n",
    "某些激活函数如ReLU也具有一定的正则化作用。\n",
    "\n",
    "所以隐藏层的激活函数对网络性能有重要影响,它实现了层与层之间的非线性变换,是网络可以拟合复杂函数的关键。\n",
    "\n",
    "* 机器学习中常用的激活函数\n",
    "\n",
    "机器学习中常用的激活函数只有恒等函数（identity function），阶跃函数（sign），sigmoid\n",
    "函数，ReLU，tanh，softmax这六种，其中Softmax与恒等函数几乎不会出现在隐藏层上，Sign、Tanh\n",
    "几乎不会出现在输出层上，ReLU与Sigmoid则是两种层都会出现，并且应用广泛。在这里，我们将总结性声明一下输出层的g(z)与隐藏层\n",
    "的h(z)之间的区别，以帮助大家获得更深的理解：\n",
    "\n",
    "1. 虽然都是激活函数，但隐藏层和输出层上的激活函数作用是完全不一样的。输出层的激活函数\n",
    "是为了让神经网络能够输出不同类型的标签而存在的。其中恒等函数用于回归，sigmoid函数用于\n",
    "二分类，softmax用于多分类。换句说， 仅仅与输出结果的表现形式有关，与神经网络的效果\n",
    "无关，也因此它可以使用线性的恒等函数。但隐藏层的激活函数就不同了，如我们之前尝试的\n",
    "XOR，隐藏层上的激活函数 的选择会影响神经网络的效果，而线性的 是会让神经网络的结\n",
    "构失效的。\n",
    "\n",
    "2. 在同一个神经网络中，输出层与隐藏层的激活函数可以是不同的，并且在大多数运行回归和多分类的神经网络时，他们也的确是不同的。每层上的 可以是不同的，但是同一层上的激活函数必须一致。\n",
    "\n",
    "* ReLU\n",
    "\n",
    "ReLU(Rectified Linear Unit)是深度学习中最常用的激活函数之一,它的定义是:\n",
    "\n",
    "$$ f(x) = \\max(0, x) $$\n",
    "\n",
    "即对输入x:\n",
    "\n",
    "如果x>0, 输出x\n",
    "\n",
    "如果x<=0, 输出0\n",
    "\n",
    "ReLU激活函数的主要特点包括:\n",
    "\n",
    "解决了梯度消失问题,因为其导数为1或0,不会产生梯度消失。\n",
    "计算简单,仅仅是一个阈值判定,计算速度快。\n",
    "在一定程度上引入了非线性,但保留了线性非饱和区。\n",
    "比Sigmoid和Tanh收敛速度更快,模型训练效率高。\n",
    "使模型更稀疏,有利于特征提取。\n",
    "但也存在死亡神经元(dead neuron)问题,即部分神经元可能永远不会活化。\n",
    "ReLU是目前最常用和最有效的激活函数之一。\n",
    "综上,ReLU激活函数通过解决梯度消失问题,提高了深度网络的训练效率,被广泛应用在深度神经网络中。但也有一些缺点需要注意。\n",
    "\n",
    "\n",
    "* 前向传播的标准实现形式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "x=torch.tensor(range(-10,10),dtype = torch.float32)\n",
    "y=torch.relu(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1bd047b5040>]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAooAAAEYCAYAAADbMtdZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAAxOAAAMTgF/d4wjAAAr/UlEQVR4nO3deXxU5aH/8e/MZJIwCVsSNsnGrmxhSUCk1q22t1xcWmtR1AQJuFS9ctH219ZrbW2rvdWiIq0iJkqgUESwarXaYrVqXSDsm8o2IQNhCQlL9snM8/vD2yhmgAEyObN83q9X/kjOIc/3HJ6EL2fmOcdmjDECAAAAvsJudQAAAACEJ4oiAAAAAqIoAgAAICCKIgAAAAKiKAIAACAgiiIAAAACoigCAAAgoLhQfvOEhAR169YtlEMAAADgLBw8eFCNjY0Bt4W0KHbr1k0ejyeUQwAAAOAspKenn3AbLz0DAAAgIIoiAAAAAqIoAgAAICCKIgAAAAKiKAIAACCgkK56BgAAQHCMMSotq5a7slbZaUnKzeoqm81maSaKIgAAgMU81XXKL16p8qo6OR12eX1+ZaS4VDJ1jNK7uizLxUvPAAAAFjLGKL94pcoO1cnrM6pr8snrMyo7VKeC4pUyxliWjaIIAABgodKyanmq6uXzH18IfX6j3VV1Ki2rtigZRREAAMBS7spa2U/wVkSnwy53ZW37BvoSiiIAAICF3Idq1dDsD7jN6/MrOy2pnRN9gcUsAAAAFvD7jR7526d66p0dirPbZIyR70uvPjvsNmWmuJSb1dWyjFxRBAAAaGe1jc26beFqPfXODo3M7KJlPxinrLQkOR02ueIdcjpsyk51qaRwrKW3yOGKIgAAQDvae7he0+aXakvFUV2Zc45++73hSnQ69NbMi7iPIgAAQKxaV35Y00tKdfBYo2ZePlB3Xdq/pQzabDblZacoLzvF4pRfoCgCAAC0g1fX79W9S9dLkuZMHqmJw8+xONGpURQBAABCyBijJ97apsdXbFP3jgmal5+rnIwuVscKCkURAAAgRBq8Pv3wxQ16df1eDTmnk54tyFWvzh2sjhU0iiIAAEAIHDjWoFtKVmtd+WH9x5CemjUpR674yKpekZUWAAAgAmzZe1TT5q/S3iMNuuOSfrrn8kGyn+jxK2GMoggAANCG/rZ5n2YsWadmn9Gs7+fou6PSrY50xiiKAAAAbcAYo7nv7tT/vvGJUlzxmjt1tHLD6FY3Z4KiCAAAcJYam32676VNenG1RwN7JKuoIE8ZKS6rY501iiIAAMBZqKpt0m0LVmulu0qXDOqm2dePVMdEp9Wx2gRFEQAA4Axt239MU+evUnlVvaaO76P7/vM8OSJw0cqJUBQBAADOwDufHtBdi9aq3uvTQ98ZpsljM62O1OYoigAAAKfBGKP5H7j14F+2qGOiUyVTx+iC/mlWxwoJiiIAAECQvD6/fvHqZi38aLf6piWpaEqe+qQlWR0rZCiKAAAAQThS59Udi9bo/e2VGt8/VX+YPFqdXdGxaOVEKIoAAACnsKuyVoXPr9LOylrdMDZTP79yiJwOu9WxQi7oovj666/rf/7nf+T3+9Xc3Kwf/vCHKigoCGU2AACAdmeMUWlZtdyVtcpOS1JTs08/+ONaHWvw6oErBmvKBdmy2aJnZfPJ2Iwx5lQ7GWOUmpqqd955R8OHD5fb7da5556rgwcPqmPHjif8c+np6fJ4PG0aGAAAIFQ81XXKL16p8qo6OR12NTb75fMbueId+v0No3TJoO5WR2xzJ+trQV9RtNlsOnz4sCTp6NGjSk1NVUJCQpsEBAAAsJoxRvnFK1V2qE4+v5HX52vZlpoUr4sHdrMwnTWCKoo2m01LlizRd7/7XSUlJam6ulrLly9XfHx8qPMBAAC0i9Kyanmq6uXzt36xdd/RBpWWVSsvwp/dfLqCehdmc3OzfvWrX2n58uUqKyvTW2+9pZtuukmVlZXH7Tdr1iylp6e3fNTU1IQkNAAAQFtzV9bqROtTnA673JW17RsoDARVFNetW6e9e/fq61//uiQpLy9P6enpWrt27XH7zZw5Ux6Pp+UjOTm57RMDAACEQGOzT/Vef8BtXp9f2VF8v8QTCaooZmRkqKKiQlu3bpUkbd++XTt27NCgQYNCGg4AAKA9LF/j0S9e2SKbpK8+qtlhtykzxaXcrK6WZLNSUO9R7NGjh5555hl9//vfl91ul9/v15w5c5SZGX3PNAQAALHD7zd69G+f6g/v7FCvzol66DtD9cvXtrasevb6/MpMcamkcGzM3BLny4K6Pc6Z4vY4AAAgXNU1NWvmkvV6Y/M+5WR00bybRqt7p8RW91HMzeoa1SWxTW6PAwAAEC32HWnQtJJV2rTnqCYO76VHr81RotMh6fO7veRlp8TcCudAKIoAACCmbPAc1rT5pTpwrFF3XzZAM74xIKqvGJ4NiiIAAIgZr2+s0MwX1skYafb1I3VlzjlWRwprFEUAABD1jDGa84/t+t3fP1O3jgmal5+rERldrI4V9iiKAAAgqjV4ffp/yzbo5XV7NbhXJz1bkKtzunSwOlZEoCgCAICodeBYg24pWa115Yf1zcE99NikEUpKoP4EizMFAACi0taKoyp8fpX2HmnQbRf104++NUj2r95NGydFUQQAAFFnxZb9+q8/rZXX59cj3xuua3MzrI4UkSiKAAAgahhjNO+9nXr4r5+oSwennr95jMb04X6IZ4qiCAAAokJTs1//8+eNeqHUowHdk1VUkKfMVJfVsSIaRREAAES86tom3bZwtT7eVaWLBnbTk5NHqlOi0+pYEY+iCAAAItr2A8dUOL9UZYfqdPP4bN034TzFOexWx4oKFEUAABCx3v3soO5YtEZ1TT79+jtDdcPYLKsjRRWKIgAAiEjzP3Drwb9sUVK8QyVTx2h8/zSrI0UdiiIAAIgozT6/fvHqFi34qEx90pL0bEGu+nVLtjpWVKIoAgCAiHGk3qs7F63Re9sqNa5vqp66cZS6uOKtjhW1KIoAACAsGWNUWlYtd2WtstOSlJYUr2klpdpxsFbXj8nQg1cNlZNFKyFFUQQAAGHHU12n/OKVKq+qk9NhV1OzX35jZIx0/8TBmjo+WzYbj+MLNYoiAAAIK8YY5RevVNmhOvn8Rl6fr2Vbr86JlMR2xPVaAAAQVkrLquWpqpfPb1ptq6xpVGlZtQWpYhNFEQAAhBV3Za0cjsBXDJ0Ou9yVte2cKHZRFAEAQFhxJTjU0OQLuM3r8ys7LamdE8UuiiIAAAgbq8uq9bM/b5KR9NW3ITrsNmWmuJSb1dWSbLGIoggAAMLCn9fu0fXzPlJdk1+/vnqI+qQlyemwyRXvkNNhU3aqSyWFY1nI0o5Y9QwAACzl9xs9tuIzPfmP7erZKVHPFuRqaO/Omjw267j7KOZmdaUktjOKIgAAsEx9k0/3LF2n1zfu0/D0zpqXn6senRIlSTabTXnZKcrLTrE4ZeyiKAIAAEvsO9Kg6SWl2rjniP5zeC89+r0cdYh3WB0LX0JRBAAA7W6j54imlazS/qON+q/LBmjGZQNkt/OycrihKAIAgHb1140V+u8X1slvpCeuG6GrRvS2OhJOgKIIAADahTFGv397ux7922dKS07QM/mjNSqTW92EM4oiAAAIuQavTz9ZvlEvrd2jc3t2VNGUPPXu0sHqWDgFiiIAAAipyppG3bpgtVaXVesb53XX49eNVHICFSQS8LcEAABC5pN9R1X4fKn2HK7XrV/vqx/9x7lysGglYlAUAQBASPzjk/26a9FaNfn8+u01w/X9vAyrI+E0URQBAECbMsao6P1d+vXrW9Wlg1PFU/I0tm+q1bFwBiiKAACgzTQ1+/WzlzfpT6vK1b97sooKcpWVmmR1LJwhiiIAAGgT1bVNuv2Pq/XRzipdOCBNv79hlDolOq2OhbNAUQQAAGdt+4EaTZu/Su5DdSoYl6X7Jw5WnMNudSycJYoiAAA4K+9vq9Ttf1ytuiaffnnVEN00LtvqSGgjFEUAAHDGFnxUpp+/slmueIeevzlPFw7oZnUktCGKIgAAOG3NPr9+9dpWPf+BW1mpLhUV5Kl/92SrY6GNBV0UGxsbdc899+jNN99UYmKicnJytHDhwlBmAwAAYcIYo9Kyarkra9WtY4KK3t+l97ZVamyfFD1942h1TYq3OiJCIOii+OMf/1g2m02fffaZbDab9u3bF8pcAAAgTHiq65RfvFLlVXWKs9tU7/VLkiYO66VZk0YoPo5FK9EqqKJYW1uroqIieTwe2WyfP3anZ8+eIQ0GAACsZ4xRfvFKlR2qk89v5PUZSZLNJm3dd1ROB4/ji2ZB/Rdgx44dSklJ0UMPPaTc3FxdeOGFeuutt0KdDQAAWKy0rFqeqnr5/Oa4rxsj7a6qU2lZtUXJ0B6CKorNzc0qKyvT4MGDVVpaqtmzZ2vSpEnav3//cfvNmjVL6enpLR81NTUhCQ0AANrHzoO18hsTcJvTYZe7sradE6E9BVUUMzMzZbfbdcMNN0iSRo4cqT59+mjjxo3H7Tdz5kx5PJ6Wj+RkVj8BABCpahubtbS0XM3+wEXR6/MrO43H80WzoIpiWlqaLrvsMr355puSpF27dmnXrl0677zzQhoOAABYY8/hen3v6Q9VWlat5IQ4OezHvxfRYbcpM8Wl3KyuFiVEewh6mdLTTz+tRx55RMOGDdPVV1+tuXPnqnfv3qHMBgAALLBmd7WumvMvba04qnu/OVB/vftCZaW65HTY5Ip3yOmwKTvVpZLCsS2LXBGdbMac4I0HbSA9PV0ejydU3x4AALSxl9ft0Q9f3CC7TZr1/RGaMKyXpOPvo5idlqTcrK6UxChxsr7Gk1kAAID8fqPH39qm2W9tU49OCZqXn6vh6V1atttsNuVlpygvO8W6kGh3FEUAAGJcfZNP9764Xq9tqNDQ3p30bH6eenZOtDoWwgBFEQCAGHbgaIOml5RqveeIvj20p373/Ry54qkH+BwzAQCAGLVpzxFNm1+qfUcbdOcl/TXz8oGy23nfIb5AUQQAIAa9sWmf/nvJOvn8Ro9NytF3RqZbHQlhiKIIAEAMMcboqX/u0G/f+FRpyfGae1OuRnMvRJwARREAgBjR2OzTT5Zv1PI1ezSoR0cVTclVeleX1bEQxiiKAADEgEM1jbp1wWqVllXr0nO7a/b1I5WcQA3AyTFDAACIcp/uO6bC+avkqa7XtK/10U8mnNfqkXxAIBRFAACi2NufHNBdi9eqwevTb747TNeNybQ6EiIIRREAgChkjFHxv9z69Wtb1KmDUwvyx2pcv1SrYyHCUBQBAIgyXp9fP3t5sxav3K2+3ZJUXJCn7LQkq2MhAlEUAQCIIofrmnT7wjX6cOchfa1/mn4/eZQ6u5xWx0KEoigCABAldh6sUeH8Uu2qrNVN52fpZ1cMltNhtzoWIhhFEQCAKPCv7ZW6feFq1TQ26xdXDlHBBdlWR0IUoCgCABDh/vhxmX728ma5nA49d/MYXTSwm9WRECUoigAARBBjjErLquWurFVG1w56Y/M+Pf9BmTJTXCoqyNWAHh2tjogoQlEEACBCeKrrlF+8UuVVdYpz2NXQ5JORlJPeWc/dPEYpSfFWR0SUoSgCABABjDHKL16pskN18vmNvD6fJMkm6Vhjs7qyshkhwFIoAAAiQGlZtTxV9fL5zXFfN5LKq+pUWlZtTTBENYoiAAARwF1Ze8JtTof9pNuBM0VRBAAgzPn9Ru9vr1STzx9wu9fn58krCAneowgAQBirbWzWfy9Zp79t2a+EOLuafUY+88XLzw67TZkpLuVmdbUwJaIVVxQBAAhTew/X69qnP9TftuzXlTnn6PW7L1RWmktOh02ueIecDpuyU10qKRwrm81mdVxEIa4oAgAQhtaVH9b0klIdPNaomZcP1F2X9pfNZtNbMy9quY9idlqScrO6UhIRMhRFAADCzKvr9+repeslSXMmj9TE4ee0bLPZbMrLTlFedopV8RBDKIoAAIQJY4yeeGubHl+xTd07Jmhefq5yMrpYHQsxjKIIAEAYaPD69MMXN+jV9Xs15JxOerYgV706d7A6FmIcRREAAIsdONqg6QtWa335YX1rSA89NmmEXPH8Ew3rMQsBALDQ5r1HNG1+qSqONOgHF/fTvd8cJLudxSkIDxRFAAAs8rfN+zRjyTo1+4x+d22OrhmdbnUk4DgURQAA2pkxRnPf3an/feMTpbjiNXfqaOWyihlhiKIIAEA7amz26b6XNunF1R4N7JGsooI8ZaS4rI4FBERRBACgnVTVNum2Bau10l2lSwZ10+zrR6pjotPqWMAJURQBAGgH2/Yf09T5q1ReVa+p4/vovv88Tw4WrSDMURQBAAixdz49oLsWrVW916eHvjNMk8dmWh0JCApFEQCAEDHGaP4Hbj34ly3qmOhUydQxuqB/mtWxgKBRFAEACAGvz69fvLpZCz/arb5pSXq2IFd9uyVbHQs4LRRFAADa2JE6r+5YtEbvb6/U+P6p+sPk0ersYtEKIg9FEQCANrSrslaF81dp58Fa3TA2Uz+/coicDrvVsYAzQlEEAOAMGGNUWlYtd2WtstOSlJvVVR/uPKTbF67RsQavHrhisKZckC2bjZXNiFynXRSfe+45TZ06VS+99JKuvvrqEEQCACC8earrlF+8UuVVdXI67PL6/Orcwanq2iZ1iI9T0ZQ8XTKou9UxgbN2WkXR7XZr3rx5Ov/880OVBwCAsGaMUX7xSpUdqpPPb+T1+SRJlTVNirPbtOz2cRrUs5PFKYG2EfSbJvx+v6ZNm6Ynn3xSCQkJocwEAEDYKi2rlqeqXj6/abXNZpOONjRbkAoIjaCL4qxZszR+/HiNHj36pPukp6e3fNTU1LRJSAAAwoW7slZxjsDvO3Q67HJX1rZzIiB0gnrpedOmTVq2bJnefffdk+43c+ZMzZw5s+Xz9PT0s0sHAECYyU5LUlOzP+A2r8+v7LSkdk4EhE5QRfG9996T2+3WgAEDJEn79u3TLbfcooqKCt1+++0hDQgAQDgpr6pVc4CXnR12mzJTXMrN6mpBKiA0bMaY1rP9FC6++GLNmDHjlKue09PT5fF4zjQbAABhw+83+t3fP9Xv396hbh0TlBBn1/6jDS2rnjNTXCopHKveXTpYHRU4LSfra9xHEQCAU6hratY9L6zXXzftU05GF827abS6dUxodR9F7pmIaHNGVxSDxRVFAECk23ekQdNKVmnTnqOaOLyXHr02R4lOh9WxgDbDFUUAAM7ABs9hTZtfqgPHGnX3ZQM04xsDuGqImEJRBAAggNc3VmjmC+tkjDT7+pG6MuccqyMB7Y6iCADAlxhjNOcf2/W7v3+mtOQEzcsfrZGZrGRGbKIoAgDwfxq8Pv2/ZRv08rq9Oq9XJxUV5OocVjEjhlEUAQCQdPBYo25ZUKq1uw/rm4N76LFJI5SUwD+TiG38BAAAYt7WiqOaNr9Uew7X67aL+ulH3xoku51FKwBFEQAQ01Zs2a+7/7RWTT6/HvnecF2bm2F1JCBsUBQBADHJGKNn39ulh/66VV06OPXczWM0pk+K1bGAsEJRBADEnKZmv+7/8yYtKS3XgO7JKirIU2aqy+pYQNihKAIAYkp1bZNuW7haH++q0kUDu+nJySPVKdFpdSwgLFEUAQAxY/uBGhXOX6WyQ3W6eXy27ptwnuIcdqtjAWGLoggAiAnvfnZQdyxao7omn379naG6YWyW1ZGAsEdRBABEvZIP3frFq1uUFO9QydQxGt8/zepIQESgKAIAolazz68H/7JFJR+WqU9akp4tyFW/bslWxwIiBkURABCVjtR7deeiNXpvW6XG9U3VUzeOUhdXvNWxgIhCUQQARDxjjErLquWurFV2WpK6JcercH6pdhys1fVjMvTgVUPlZNEKcNooigCAiOaprlN+8UqVV9XJ6bCrqdkvvzEyRrp/4mBNHZ8tm43H8QFngqIIAIhYxhjlF69U2aE6+fxGXp+vZVuvzomUROAscR0eABCxSsuq5amql89vWm2rrGlUaVm1BamA6EFRBABELHdlrRyOwFcMnQ673JW17ZwIiC4URQBAxEpKiFNDky/gNq/Pr+y0pHZOBEQXiiIAICKt2V2t+/+8UUbSV9+G6LDblJniUm5WV0uyAdGCoggAiDgvr9uj6575SHVNfv366iHqk5Ykp8MmV7xDTodN2akulRSOZSELcJZY9QwAiBh+v9HjKz7T7H9sV89OiXq2IFdDe3fW5LFZx91HMTerKyURaAMURQBARKhv8unepev12sYKDU/vrHn5uerRKVGSZLPZlJedorzsFItTAtGFoggACHv7jzZoekmpNniO6D+H9dKj1+aoQ7zD6lhA1KMoAgDC2qY9RzRtfqn2HW3Qf102QDMuGyC7nZeVgfZAUQQAhK03NlXov5esl88YPXHdCF01orfVkYCYQlEEAIQdY4z+8M4OPfLmp0pLTtAz+aM1KpNb3QDtjaIIAAgrjc0+/WTZRi1fu0fn9uyooil56t2lg9WxgJhEUQQAhI3KmkbdumC1VpdV6xvnddfj141UcgL/VAFW4acPABAWPt13TIXzV8lTXa9bv95XP/qPc+Vg0QpgKYoiAMByb39yQHctXqvGZp9+e81wfT8vw+pIAERRBABYyBijovd36aHXt6pTB6eeLRir8/umWh0LwP+hKAIALNHU7NcDr2zS4pXl6tctScVT8pSVmmR1LABfQlEEALS7w3VNum3han20s0oXDkjTnMmj1LmD0+pYAL6CoggAaFc7Dtao8PlVch+qU/64LP1s4mDFOexWxwIQAEURANBu3t9WqR/8cbVqm3x68Kohyh+XbXUkACdBUQQAtIuFH5XpgVc2yxXv0PM35+nCAd2sjgTgFCiKAICQavb59avXtur5D9zKSnWpqCBP/bsnWx0LQBCCKooNDQ267rrrtGXLFnXo0EHdu3fXU089pf79+4c6HwAgghhjVFpWLXdlrbLTkjSwR7L+a/E6/fOzgxrbJ0VP3zhaXZPirY4JIEhBX1G85ZZb9O1vf1s2m01z5szRtGnT9M4774QwGgAgkniq65RfvFLlVXVyOuzy+vySJK/PaFJuhn559VDFx7FoBYgkQf3EJiYmasKECbLZPn+U0vnnny+32x3KXACACGKMUX7xSpUdqpPXZ1TX5JPXZ+T1GaUmxevh71ISgUh0Rj+1TzzxhK666qq2zgIAiFClZdXyVNXL5zetth1t8Gr17sPtHwrAWTvtovjQQw9p+/btevjhh1ttmzVrltLT01s+ampq2iQkACC8uStrFeewBdzmdNjlrqxt50QA2sJpFcVHH31Uy5cv11//+le5XK5W22fOnCmPx9PykZzMqjYAiAU9OiWq3usLuM3r8ys7jUfzAZEo6MUss2bN0uLFi7VixQp16dIlhJEAAJFk7+F6PfT6Vhkj2SR9+cVnh92mzBSXcrO6WhUPwFmwGWNav6HkKzwejzIyMtS3b1917NhRkpSQkKCPP/74pH8uPT1dHo+nbZICAMLOuvLDml5SqoPHGjX9wj56a+sBlVd/seo5M8WlksKx6t2lg9VRAZzAyfpaUFcU09PTFUSfBADEkFfW79UPl66XzSb94YZRmjCsl3464bzj7qOYm9W15Y4ZACIPT2YBAJwWY4weW7FNs9/aph6dEjQvP1fD07tIkmw2m/KyU5SXnWJtSABtgqIIAAhag9ene5au12sbKjS0dyc9m5+nnp0TrY4FIEQoigCAoBw42qDpJaVa7zmi/xjSU7Mm5cgVzz8jQDTjJxwAcEqb9hzR9JJSVRxp0B2X9NM9lw+S3c57D4FoR1EEAJzUm5v3acaf1snnN3psUo6+MzLd6kgA2glFEQAQkDFGT/9zp3775idKccXrmfzRGp3FIhUgllAUAQCtNDb79NPlm7RsjUeDenTUswW5ykhp/UQuANGNoggAOM6hmkbdumC1Ssuqdem53fXEdSPUMdFpdSwAFqAoAgBafLb/mKY+v0qe6noVfq2PfjrhPDlYtALELIoiAECS9PanB3TXorVq8Pr08HeH6foxmVZHAmAxiiIAxDhjjJ77l1u/em2LOiY6VVI4Rhf0S7M6FoAwQFEEgBjm9fn1wCubtejj3eqblqSiKXnqk5ZkdSwAYYKiCAAx6kidVz9YtFr/2n5IX+ufpt9PHqXOLhatAPgCRREAYtCuyloVPr9KOytrdeP5mXrgiiFyOuxWxwIQZiiKABBjPtheqdv/uEbHGrz6+RWDVXBBtmw2VjYDaI2iCAAxZNHHu/Wzlzepg9Oh4il5unhQd6sjAQhjFEUAiELGGJWWVctdWavstCSNzOiiX7++Vc/9y62MlA4qLsjTgB4drY4JIMxRFAEgyniq65RfvFLlVXVyOuzy+vyKs9tV7/UpL7urnr5xtFKTE6yOCSACUBQBIIoYY5RfvFJlh+rk8xt5fT5JktfnU8eEOC0oHKNEJ7/6AQSHJW4AEEVKy6rlqaqXz29abWto9mnjnqMWpAIQqSiKABBF3JW1inMEXsHsdNjlrqxt50QAIhlFEQCiSGaKSw1eX8BtXp9f2Tx1BcBpoCgCQJSoa2pW0fs7FeBVZznsNmWmuJSb1bX9gwGIWLyjGQCiQMWRehU+X6otFUd12XndtfNgrTzVX6x6zkxxqaRwLDfWBnBaKIoAEOHWlx/WtJJSHTzWqBnfGKC7LxsgScfdRzE3qyslEcBpoygCQAT7y4a9uueF9ZKkJ68fqStyzmnZlpedorzsFKuiAYgCFEUAiEDGGM1+a7seW/GZunVM0Lz8XI3I6GJ1LABRhqIIABGmwevTj17coFfW79XgXp1UNCVXvTp3sDoWgChEUQSACHLgWINuKVmtdeWH9c3BPfT4dSPkiudXOYDQ4LcLAESILXuPatr8Vdp7pEG3X9xPP/zmINntLFABEDoURQCIAH/fsl93/2mtvD6/Hr02R98bnW51JAAxgKIIAGHMGKO57+7U/77xibq64jV/6hhWMgNoNxRFAAhTTc1+3ffSRi1d7dHAHskqKshTRorL6lgAYghFEQDCUFVtk25bsFor3VW6eFA3PXn9SHVMdFodC0CMoSgCQJjZtv+YCueXandVnaaO76OfTjhXcQ671bEAxCCKIgCEkX9+dlB3/nGN6r0+/fo7Q3XD2CyrIwGIYRRFAAgT8z9w6xevblbHRKdKpo7RBf3TrI4EIMZRFAHAYl6fXw++ukULPipT37QkPVuQq77dkq2OBQAURQCw0pE6r+5YtEbvb6/UBf1S9dQNo9XZxaIVAOGBoggAFtlVWavC+au082CtJo/N1C+uHCIni1YAhBGKIgBY4MMdh3TbwtU61uDVzyYO1s3js2Wz8Tg+AOEl6KK4bds2FRQUqLKyUp07d9bzzz+vIUOGhDJb0IwxKi2rlruyVtlpScrN6touv3CtGDeWjtWqcWPpWK0aN5aONdC42/cf0/0vb1ai06GiKXm6ZFD3kGcAgDMRdFG89dZbdcstt2jKlCl68cUXNWXKFK1atSqU2YLiqa5TfvFKlVfVyemwy+vzKyPFpZKpY5TeNXRPMLBi3Fg6VqvGjaVjtWrcWDrWQOM2eH3yG6lX5wTNnzpWA3t0DNnYAHC2bMYYc6qdDhw4oP79+6uqqkpxcXEyxqhXr156//331b9//xP+ufT0dHk8njYN/GXGGF02658qO1Qnn/+Lw3DYbcpOdWnFzItCcrXAinFj6VitGjeWjtWqcWPpWE82riRlp7r09r0X83IzAMudrK8FVRRXr16tyZMn69NPP2352pgxY/Sb3/xGl1566RkN3BZWuat0w7yP1eTzt9pmk5SZ6pIrvu3fhlnX1Kzdh+oU6MSFalwrxoy1cWPpWK0aN5aO9VTjOh02LZp+vvKyU9p8XAA4HSfra236m3HWrFmaNWtWy+c1NTVt+e1bcVfWKs5hU5Mv8PaahmaF4v/qNQ3Np9ze1uNaMWasjRtLx2rVuLF0rKca1+mwy11ZS1EEENaCKooZGRmqqKhQc3Nzy0vPu3fvVmZm5nH7zZw5UzNnzmz5PD09vW3TfkV2WpK8Aa4mSlKcw6anbxodkl/Cq9xVmjzvI3l9ra8ThGpcK8aMtXFj6VitGjeWjvVU43p9fmWnJbX5mADQloK6YVf37t01atQoLVy4UJK0bNkypaenn/T9ie0hN6urMlJcctiPvxbgsNuUmeJSblbXqBk3lo7VqnFj6VitGjeWjtXKcQGgrQR9Z9e5c+dq7ty5GjhwoH7zm9/oueeeC2WuoNhsNpVMHaOsVJecDptc8Q45HZ+/Ob2kcGzI3iRuxbixdKxWjRtLx2rVuLF0rFaOCwBtJajFLGcq1ItZ/i1c7o3G/eeiY9xYOlarxo2lY7VyXAAIxlmveg7FwAAAALDeyfoaDxUFAABAQBRFAAAABERRBAAAQEAURQAAAAREUQQAAEBAIV31nJCQoG7duoXq27dSU1Oj5OTkdhsvUnGegsN5Ch7nKjicp+BwnoLDeQoO5+nUDh48qMbGxoDbQloU2xu34wkO5yk4nKfgca6Cw3kKDucpOJyn4HCezg4vPQMAACAgiiIAAAACiqqiOHPmTKsjRATOU3A4T8HjXAWH8xQczlNwOE/B4Tydnah6jyIAAADaTlRdUQQAAEDboSgCAAAgoIgqiq+99ppGjx6thIQEzZgx47htfr9fd911l/r166f+/ftrzpw5J/w+27Zt0wUXXKCBAwcqLy9PmzdvDnFya91xxx0aMWJEy0diYqJmz54dcN+LL75Yffr0adn3sccea+e01vn5z3+ubt26tRz7DTfccMJ9Y20Ofdns2bM1dOhQDRs2TMOHD9fChQtPuG8szqdg50ZRUZEGDBigfv36afr06fJ6ve2c1DoNDQ26+uqrNXDgQOXk5Ojyyy/X9u3bW+3ndrvlcDiO+/21Y8cOCxJbJzs7W4MGDWo5/iVLlgTcL5bn06FDh46bIwMHDlRcXJyqqqqO24/5dIZMBPn000/NunXrzH333Wfuvvvu47bNnz/fXHrppaa5udkcOnTIZGZmmk2bNgX8Ppdccol57rnnjDHGLF261OTm5oY4efioqKgwiYmJpqKiIuD2iy66yLz00kvtGypMPPDAA63m1YnE8hxasWKFOXz4sDHGmN27d5vU1FSzffv2gPvG4nwKZm7s3LnT9OrVy1RUVBi/32+uuOIKM2fOnHZOap36+nrz2muvGb/fb4wx5sknnzQXXXRRq/127dplOnfu3L7hwkxWVpZZu3btSfeJ9fn0VY888oiZOHFiq68zn85MRF1R/Pf/PuPi4lptW7JkiaZPny6Hw6GUlBRNmjRJixcvbrXfgQMHVFpaqhtvvFGSdM0116i8vDzg/2aj0fz58/Wtb31LPXv2tDpKxIr1OXTZZZepc+fOkqSMjAz17NlT5eXlFqcKD8HOjRdffFFXXnmlevbsKZvNpttuuy3g76tolZiYqAkTJshms0mSzj//fLndbmtDRbBYn09fVVRUpMLCQqtjRI2IKoons3v3bmVlZbV8np2drd27d7far7y8XL169WopmzabTZmZmQH3jUbFxcWn/AH68Y9/rGHDhmnSpEnauXNnOyULD0uXLlVOTo4uvfRSvf322wH3ifU59GUrVqxQdXW18vLyTrhPLM2nYOdGsL+vYsUTTzyhq666KuC22tpa5eXladSoUXrwwQfl8/naOZ318vPzNWzYMBUWFurgwYOttjOfvvDBBx+ourpaEydODLid+XT6wqoojhs3TmlpaQE/uGJxYsGet/fee0/Hjh3ThAkTTvi9FixYoE8++UQbNmzQhRdeeMIftkh0qvN02223ye12a/369frlL3+pSZMmqayszOrY7S7Y+bRx40bdfPPNWrJkiZKSkgJ+r2ieT2gbDz30kLZv366HH3641bZevXppz549WrVqlVasWKH33ntPv/vd7yxIaZ13331XGzZs0Jo1a5SWlqaCggKrI4W1oqIi5efnB3zlkfl0ZlqfSQt9+OGHZ/xnMzMzVVZWpnHjxkn6/E2rmZmZrfbLyMhQRUWFmpubFRcXJ2OMdu/eHXDfSBHseSsqKlJBQYEcDscJ98nIyJD0+ZWQO++8U/fee68OHTqk1NTUNslqpdOZX+PHj9fIkSNVWlp63P/UpeicQ18WzHnasmWLJk6cqOLiYn3ta1874X7RPJ8CCXZuZGZmHvcm+hP9vop2jz76qJYvX64VK1bI5XK12p6QkKDu3btLklJSUjR16lQtWrRIP/rRj9o7qmX+PS+cTqdmzJihgQMHBtyH+STV1NTohRde0KpVqwJuZz6dmbC6ong2rr32Ws2bN08+n09VVVVasmSJJk2a1Gq/7t27a9SoUS0rNZctW6b09HT179+/vSO3q6NHj+rFF1/U1KlTT7hPc3Oz9u/f3/L5smXL1KNHj6j9R/2rvvzQ+G3btmndunUaNmxYq/1idQ7929atWzVhwgQ988wzuvzyy0+4XyzOp2DnxjXXXKNXXnlF+/btkzFGTz/9tK677jorIltm1qxZWrx4sf7+97+rS5cuAfc5cOBAy+rdxsZGLV++XCNHjmzHlNaqra3V4cOHWz5fvHhxwONnPn1uyZIlysnJ0bnnnhtwe6zPpzNm6VKa07RixQrTu3dv07FjR5OcnGx69+5tXn75ZWOMMc3NzeYHP/iB6dOnj+nbt695/PHHW/7cyy+/bAoLC1s+/+STT8z5559vBgwYYEaPHm02bNjQ7sfS3ubOnWu+/vWvt/r6qlWrzLe//W1jjDE1NTVm9OjRZujQoWb48OHm0ksvNevWrWvvqJbJz883Q4YMMTk5OWbUqFFm6dKlLduYQ1/4xje+Ybp06WJycnJaPt544w1jDPPJmBPPjcLCwpbfV8YY88wzz5i+ffuavn37mqlTp5qmpiarIre78vJyI8n07du3ZQ6NGTPGGGPM/fffb5566iljjDHLli0zQ4YMMcOHDzeDBw82d955p2loaLAyervasWOHGTFihBk2bJgZOnSoufLKK82uXbuMMcynQMaNG2eKi4uP+xrz6ezxCD8AAAAEFDUvPQMAAKBtURQBAAAQEEURAAAAAVEUAQAAEBBFEQAAAAFRFAEAABAQRREAAAABURQBAAAQEEURAAAAAf1/5B2og4m01UAAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 800x320 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize = (10,4),dpi = 80)\n",
    "plt.plot(x,y,\"o-\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 下面是一个标准torch神经网络定义例子，请尝试了解此例子，把前面我们的代码参照此例实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([ 1.3508e-01,  1.5439e-01, -1.9350e-01, -6.8777e-02,  1.3787e-01,\n",
       "        -1.8474e-01,  1.2763e-01,  1.8031e-01,  9.5152e-02, -1.2660e-01,\n",
       "         1.4317e-01, -1.4945e-01,  3.4258e-05], requires_grad=True)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch.nn as nn\n",
    "from torch.nn import functional as F\n",
    "\n",
    "torch.manual_seed(420)\n",
    "X = torch.rand((500,20),dtype=torch.float32)\n",
    "y = torch.randint(low=0,high=3,size=(500,1),dtype=torch.float32)\n",
    "\n",
    "#继承nn.Modules类来定义神经网路的架构\n",
    "class Model(nn.Module):\n",
    "#init：定义类本身，__init__函数是在类被实例化的瞬间就会执行的函数\n",
    "    def __init__(self,in_features=10,out_features=2):\n",
    "        super(Model,self).__init__() #super(请查找这个类的父类，请使用找到的父类替换现在的类)\n",
    "        self.linear1 = nn.Linear(in_features,13,bias=True) #输入层不用写，这里是隐藏层的第一层\n",
    "        self.linear2 = nn.Linear(13,8,bias=True)\n",
    "        self.output = nn.Linear(8,out_features,bias=True)\n",
    "\n",
    "#__init__之外的函数，是在__init__被执行完毕后，就可以被调用的函数\n",
    "    def forward(self, x):\n",
    "        z1 = self.linear1(x)\n",
    "        sigma1 = torch.relu(z1)\n",
    "        z2 = self.linear2(sigma1)\n",
    "        sigma2 = torch.sigmoid(z2)\n",
    "        z3 = self.output(sigma2)\n",
    "        sigma3 = F.softmax(z3,dim=1)\n",
    "        return sigma3\n",
    "        \n",
    "input_ = X.shape[1] #特征的数目\n",
    "output_ = len(y.unique()) #分类的数目\n",
    "#实例化神经网络类\n",
    "torch.manual_seed(420)\n",
    "net = Model(in_features=input_, out_features=output_)\n",
    "#在这一瞬间，所有的层就已经被实例化了，所有随机的w和b也都被建立好了\n",
    "#前向传播\n",
    "net(X)\n",
    "net.forward(X)\n",
    "#查看输出的标签\n",
    "sigma = net.forward(X)\n",
    "sigma.max(axis=1)\n",
    "#查看每一层上的权重w和截距b\n",
    "net.linear1.weight\n",
    "net.linear1.bias"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[ 1.3656e-01, -1.3459e-01,  2.1281e-01, -1.7763e-01, -6.8218e-02,\n",
       "         -1.5410e-01,  1.7245e-01,  8.3885e-02, -1.1153e-01, -1.7294e-01,\n",
       "         -1.2947e-01, -4.3138e-02, -1.1413e-01,  1.6295e-01, -9.4082e-02,\n",
       "         -1.4629e-01, -6.8982e-02, -2.1836e-01, -1.0859e-01, -1.2199e-01],\n",
       "        [ 4.8127e-02,  1.8186e-01,  2.4149e-02, -1.3032e-01,  9.2056e-02,\n",
       "         -9.5202e-02, -1.0584e-01, -4.2852e-02, -1.1669e-01,  2.4581e-02,\n",
       "          1.8152e-01,  3.0500e-02,  1.3506e-01, -1.9425e-01, -1.7591e-01,\n",
       "         -2.9751e-02,  2.0485e-04,  1.3957e-01, -1.9666e-01,  9.3293e-02],\n",
       "        [-1.9192e-01,  3.6070e-02,  1.4778e-01,  3.0845e-02,  7.1393e-02,\n",
       "          1.4217e-01,  2.2122e-01, -1.4032e-01,  7.3255e-02,  1.8409e-01,\n",
       "          1.2716e-01, -2.0253e-01, -1.5509e-01, -2.1899e-01,  9.8980e-02,\n",
       "          2.2123e-01, -2.1659e-01,  1.7880e-01, -2.0922e-01, -2.7275e-02],\n",
       "        [ 1.8144e-01, -3.5166e-02,  2.4801e-02,  1.6299e-01, -1.8755e-01,\n",
       "          5.6587e-02, -1.0911e-01,  2.0523e-01, -1.9378e-01,  1.6899e-02,\n",
       "          1.3966e-01, -1.3137e-01, -1.3201e-01,  7.6554e-02, -1.7558e-01,\n",
       "          1.3096e-01,  2.7182e-02, -2.2010e-01,  7.6883e-02, -1.8731e-01],\n",
       "        [ 2.7419e-02,  1.3699e-01, -3.8687e-02,  8.3463e-02, -1.5634e-02,\n",
       "         -1.6781e-01, -2.1426e-01,  1.8463e-01,  8.3891e-02,  5.9950e-02,\n",
       "         -2.0538e-01, -2.7832e-02,  4.7442e-02, -1.9782e-01, -1.7842e-01,\n",
       "          1.1362e-01,  1.4101e-01, -1.3794e-01,  1.1704e-01, -3.4108e-02],\n",
       "        [ 3.8388e-02, -1.7268e-01, -1.0235e-01, -1.2634e-01, -1.1883e-01,\n",
       "         -1.3463e-01, -1.7610e-01,  3.6543e-02, -1.7834e-01, -1.6471e-01,\n",
       "          2.0834e-01,  1.8400e-01, -8.8723e-02, -7.5378e-02,  1.7877e-01,\n",
       "         -5.7259e-02, -2.4522e-02, -1.1822e-02, -1.8196e-01,  1.9812e-01],\n",
       "        [-2.2011e-02,  2.1847e-01,  1.8410e-01,  9.7177e-02, -5.0634e-03,\n",
       "         -2.4731e-03,  5.1408e-03, -2.1733e-01, -5.3375e-02, -1.0346e-01,\n",
       "         -1.3303e-02,  2.7354e-02, -1.7523e-01,  1.6994e-01,  1.8259e-01,\n",
       "          1.3907e-01,  1.0041e-01,  3.5377e-02, -1.6114e-01,  9.0056e-02],\n",
       "        [ 7.9232e-02,  2.1614e-01, -2.1087e-01,  1.9407e-01,  1.7559e-01,\n",
       "          4.1470e-02,  7.4482e-02,  2.6737e-02, -1.7872e-02,  4.5040e-02,\n",
       "          1.2947e-01,  2.5483e-02, -2.0320e-02, -7.3942e-03, -1.7221e-01,\n",
       "         -1.0705e-01,  1.8203e-01,  1.3179e-02,  2.3468e-02, -1.9567e-01],\n",
       "        [ 1.6338e-01,  8.0209e-03, -2.9885e-02, -2.1884e-01,  1.3471e-01,\n",
       "         -2.8901e-02, -1.8757e-01,  8.9256e-03,  2.0940e-01,  9.0927e-02,\n",
       "         -8.2969e-02, -9.0893e-03,  1.0047e-01, -1.6897e-02, -1.3736e-01,\n",
       "          1.6801e-01, -1.9342e-01, -3.4822e-02,  1.0057e-01,  2.2273e-02],\n",
       "        [ 1.4611e-01,  1.4414e-01, -2.3093e-02,  8.1946e-02,  5.9792e-03,\n",
       "          6.7672e-02,  1.5254e-01,  1.6742e-01, -1.6896e-01,  1.1571e-01,\n",
       "         -1.8538e-01,  2.3316e-02, -1.6147e-01,  1.0230e-01, -1.7314e-01,\n",
       "         -1.8906e-01, -2.0286e-01, -2.1210e-02, -2.1799e-02, -3.7921e-02],\n",
       "        [ 1.9375e-01,  5.3921e-02, -1.4900e-01,  1.6709e-01, -1.6652e-01,\n",
       "          6.2363e-02, -4.1574e-02, -2.0565e-01, -1.3649e-01, -2.0600e-01,\n",
       "         -1.9032e-01, -8.8942e-02, -7.8061e-02,  1.6323e-01, -1.3174e-01,\n",
       "          5.8638e-02,  2.1117e-01,  1.6707e-01, -5.9492e-02, -2.0973e-01],\n",
       "        [-2.5644e-02, -1.0818e-02, -3.3051e-02,  3.7071e-02, -1.0809e-01,\n",
       "          2.0642e-01,  1.2396e-01, -2.1523e-01,  1.2172e-01, -1.4323e-01,\n",
       "          1.1334e-01,  4.6931e-02,  8.4553e-02,  2.0530e-01, -1.1833e-01,\n",
       "          1.9287e-01, -2.8398e-02,  7.1443e-03, -2.1055e-01,  1.0805e-01],\n",
       "        [-1.2258e-01, -6.8325e-02, -2.1929e-01, -1.4939e-01,  1.9226e-01,\n",
       "         -6.2922e-02, -7.6377e-02,  2.1955e-01, -4.5838e-02,  9.8011e-03,\n",
       "         -2.9400e-03, -9.5241e-02, -7.9775e-02, -1.8708e-01,  1.7828e-01,\n",
       "         -1.7552e-01, -1.0328e-01, -1.9697e-02, -1.7449e-01,  2.0408e-02]],\n",
       "       requires_grad=True)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.linear1.weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 神经网络的损失函数\n",
    "\n",
    "\n",
    "* b站线性回归相关讲解视频\n",
    "\n",
    "https://www.bilibili.com/video/BV1QM4y167oZ/?spm_id_from=333.788&vd_source=09af776e3fe3bbe20ff8249e4a9b1581\n",
    "\n",
    "\n",
    "### 回归问题中的损失函数一般为均方误差(Mean Squared Error, MSE)\n",
    "\n",
    "误差平方和损失函数(Mean Squared Error, MSE)是回归任务中常用的损失函数,其思想是计算预测值和真实值的平方差误差。\n",
    "\n",
    "MSE 函数的表达式为:\n",
    "\n",
    "$$ MSE=\\frac{1}{n}\\sum_{i=1}^{n}(y_{pred_{i}}-y_{true_{i}})^{2}$$\n",
    "\n",
    "其中$n$是样本数量,$y_{pred_{i}}$是预测值,$y_{true_{i}}$是真实标签。\n",
    "\n",
    "MSE的主要特点:\n",
    "\n",
    "- 易于计算,代码实现简单\n",
    "- 对大误差VERY敏感,能惩罚大的误差\n",
    "- 容易落入局部最小值\n",
    "\n",
    "在PyTorch中,MSE损失函数的实现代码如下:\n",
    "这里直接使用PyTorch内置的MSELoss计算了预测值和真实值之间的MSE。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.6667)\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "# 预测值 \n",
    "y_pred = torch.tensor([2, 3, 4], dtype = torch.float32)  \n",
    "\n",
    "# 真实值\n",
    "y_true = torch.tensor([1, 3, 5], dtype = torch.float32)\n",
    "\n",
    "# 计算 MSE\n",
    "criterion = nn.MSELoss()\n",
    "loss = criterion(y_pred, y_true)\n",
    "\n",
    "print(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 二分类问题中的交叉熵损失函数(Binary Cross Entropy, BCE)\n",
    "\n",
    "用来衡量两个概率分布之间的距离,通常用来训练二分类神经网络。\n",
    "\n",
    "其表达式为:  \n",
    "\n",
    "$$ BCE(y, \\hat{y}) = - (y \\log \\hat{y} + (1-y) \\log (1-\\hat{y})) $$\n",
    "\n",
    "这里$y$是真实标签,$ŷ$是模型预测概率。\n",
    "\n",
    "BCE的主要特点:\n",
    "\n",
    "- 输出使用Sigmoid函数,范围在[0,1]表示概率\n",
    "- 对正负样本同时起作用\n",
    "- 易于求导,便于梯度下降\n",
    "- 输出非0即1时效果最好\n",
    "\n",
    "在PyTorch中,BCE损失函数代码实现:\n",
    "\n",
    "这里直接调用PyTorch的BCELoss来计算二分类交叉熵损失。\n",
    "\n",
    "所以BCE通过概率距离度量二分类效果,实现了简单而有效的二元交叉熵损失函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.2899)\n"
     ]
    }
   ],
   "source": [
    "# 预测概率 \n",
    "y_pred = torch.tensor([0.2, 0.7])  \n",
    "\n",
    "# 真实标签\n",
    "y_true = torch.tensor([0, 1], dtype = torch.float32) \n",
    "\n",
    "# 计算BCE\n",
    "criterion = nn.BCELoss()  \n",
    "loss = criterion(y_pred, y_true)\n",
    "\n",
    "print(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 多类交叉熵损失函数(Multi-class Cross Entropy)\n",
    "\n",
    "对于多分类问题,我们通常使用多类交叉熵损失函数(Multi-class Cross Entropy)。\n",
    "\n",
    "其计算公式为:\n",
    "\n",
    "$$ BCE(y, \\hat{y}) = - \\sum_{c=1}^My_{o,c}\\log(\\hat{y}_{o,c}) $$\n",
    "\n",
    "这里$y_o$是onehot编码的真实标签,$\\hat{y}_o$是模型输出的概率,M是类别数量。\n",
    "\n",
    "多类交叉熵的工作原理是:\n",
    "\n",
    "1. 将真实标签转化为onehot向量,只有对应类别索引位置为1。\n",
    "\n",
    "2. 模型通过Softmax输出每个类别的预测概率。\n",
    "\n",
    "3. 交叉熵度量预测概率分布与onehot标签分布的距离。\n",
    "\n",
    "4. 最小化此距离等效于最大化对的类别概率。\n",
    "\n",
    "PyTorch中多类交叉熵损失函数的实现代码:\n",
    "\n",
    "所以多类交叉熵适合多分类问题,通过onehot编码 label 并最小化预测概率与真实label 分布之间的距离来训练模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.9539)\n"
     ]
    }
   ],
   "source": [
    "# 预测概率\n",
    "y_pred = torch.tensor([[0.1, 0.2, 0.7],\n",
    "                       [0.3, 0.5, 0.2]])\n",
    "# 真实标签 \n",
    "y_true = torch.tensor([2, 0]) #由于交叉熵损失需要将标签转化为独热形式，因此不接受浮点数作为标签的输入\n",
    "\n",
    "# 计算多类交叉熵  \n",
    "criterion = nn.CrossEntropyLoss()\n",
    "loss = criterion(y_pred, y_true)\n",
    "print(loss)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
